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Sensor  Data  Fusion  and  Integration 
of  the  Human  Element 

(RTO  MP-12) 

Executive  Summary 

Operators  of  future  military  systems  in  a  digitised  battlespace  will  be  faced  with  increasing  volumes  of 
data.  These  data  will  derive  from  multiple  sources  including  on-board  sensors,  other  platforms  and 
systems.  Sensor  Data  Fusion  (SDF)  aims  to  integrate  these  data  in  order  to  present  the  simplest  and 
most  accurate  picture  to  the  operator.  Perceived  benefits  include  reduced  operator  workload,  improved 
mission  management  and  improved  decision-making. 

The  prime  purpose  of  this  symposium  was  to  provide  an  opportunity  for  the  NATO  community  to 
explore  and  discuss  the  technological  and  operational  issues  raised  by  the  need  to  ensure  effective 
integration  of  SDF  systems  and  the  military  operators. 

The  symposium  was  structured  into  sessions  as  follows: 

•  Characteristics  of  Operational  Requirements 

•  System  Design  Techniques  and  Technologies 

•  Integration  of  Human  Operators  with  Complex  Systems 

•  System  Applications 

•  Lessons  learned  and  Future  Trends 

The  papers  presented  addressed  a  broad  spectrum  of  issues  for  a  variety  of  different  platforms  and 
mission  types.  The  main  areas  covered  were: 

•  Development  and  implementation  of  data  fusion  algorithms 

•  Design  of  SDF  Human  Machine  Interfaces 

•  Methods  for  evaluating  the  effectiveness  of  integrated  SDF/crew  systems 

•  Results  of  trials  assessing  the  performance  of  integrated  SDF/crew  systems 

The  issue  of  mission  responsibility  was  discussed:  was  the  human  or  the  machine  to  be  held  responsible 
for  “decisions”  made  by  an  SDF  system?  What  checks  should  systems  incorporate  to  ensure  that  the 
operator  could  verify  or  override  a  fusion  decision?  In  today’s  increasingly  politically  sensitive  climate, 
this  issue  could  be  critical  to  the  implementation  of  SDF  systems. 

There  seemed  little  doubt  amongst  the  attendees  that  SDF  systems  could  significantly  enhance  mission 
effectiveness.  However,  it  was  also  agreed  that  integration  of  the  human  element  was  vital.  The 
involvement  of  crew  in  the  design  and  prototyping  process  was  felt  to  be  key.  The  importance  of 
adequate  training  was  stressed.  Training  must  ensure  that  crew  not  only  understand  the  capabilities  of 
the  system  but  also  trust  the  information  generated. 


Dr  George  E  A  Reid 

Chairman,  Technical  Planning  Committee 


La  fusion  de  donnees  de  senseur  et 
I’integration  du  facteur  humain 

(RTO  MP-12) 


Synthese 

Les  operateurs  des  futurs  systemes  militaires  sur  un  champ  de  bataille  en  environnement  numerise 
devront  faire  face  a  des  volumes  grandissants  de  donnees.  Ces  donnees  proviendront  de  sources 
multiples  telles  que  capteurs  embarques  et  autres  plates-formes  et  systemes.  Le  fusionnement  des 
donnees  capteurs  (SDF)  permet  d’integrer  ces  donnees  afin  de  presenter  la  situation  a  I’operateur  de  la 
fagon  la  plus  simple  et  la  plus  precise  possible.  Les  avantages  escomptes  comprennent  la  diminution  de 
la  charge  de  travail  de  I’operateur,  ainsi  que  1’ amelioration  de  la  gestion  de  la  mission  et  de  la  prise  de 
decisions. 

Ce  symposium  a  eu  pour  objectif  principal  de  permettre  aux  representants  des  pays  membres  de 
rOTAN  d’examiner  et  de  discuter  des  questions  technologiques  et  operationnelles  soulevees  par 
I’integration  effective  des  systemes  SDF  dans  1’ environnement  de  travail  des  operateurs  militaires. 

Le  symposium  a  ete  organise  sous  forme  de  sessions  : 

•  Caractdristiques  des  besoins  operationnels 

•  Techniques  et  technologies  de  conception  systemes 

•  Integration  des  operateurs  humains  dans  les  systemes  complexes 

•  Applications  systemes 

•  Enseignements  tires  et  tendances 

Les  communications  presentees  ont  aborde  un  large  eventail  de  sujets  concemant  divers  types  de  plates- 
formes  et  de  missions.  Les  principaux  domaines  converts  furent  : 

•  Le  developpement  et  la  mise  en  oeuvre  d’algorithmes  de  fusionnement  de  donnees 

•  La  conception  des  interfaces  homme-machine  SDF 

•  Les  methodes  d’evaluation  de  Tefficacite  de  I’integration  des  systemes  SDF 

•  Les  resultats  des  essais  de  performance  des  systemes  SDF 

La  question  de  la  responsabilite  de  la  mission  a  ete  discutee  :  Les  «  decisions  »  prises  par  un  systeme 
SDF,  sont-elles  imputables  a  I’etre  humain  ou  a  la  machine  ?  Quels  sont  les  tests  integres  a  inclure  pour 
permettre  a  I’operateur  de  verifier  une  decision  de  fusion,  afin  d’en  tenir  compte  ou  non.  Etant  donne  la 
sensibilite  du  climat  politique  actuel,  cette  question  pourrait  etre  determinante  pour  1’ adoption  des 
systemes  SDF. 

La  majorite  des  participants  semblait  convaincus  que  1’ integration  des  systemes  SDF  pourrait  ameliorer 
de  fa9on  considerable  I’efficacite  des  missions,  en  etant  cependant  d’ accord  sur  le  fait  que  1’ integration 
de  I’element  humain  etait  indispensable  et  que  I’implication  des  membres  d’equipage  au  stade  de 
conception  demeurerait  un  element  cle.  Les  participants  ont  souligne  I’importance  d’une  formation 
appropride  qui  devrait  permettre  aux  equipages  non  seulement  d’apprecier  les  capacites  du  systeme, 
mais  aussi  d’avoir  confiance  dans  les  informations  qu’il  genere. 


Dr  George  E  A  Reid 

President  du  comite  de  planification  technique 
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Theme 


The  theme  is  the  effective  collaboration  between  crew  and  the  complex,  but  not  entirely  autonomous,  systems  in  the  context 

of  multiple  information  sources  such  as  off-board  communications,  other  platforms  and  multiple  sensors  on  board  platforms. 

The  primary  components  of  this  theme  that  merit  particular  attention  are: 

(a)  Characteristics  of  operational  requirements  demanding  difficult  human-system  interactions; 

(b)  System  design  and  technology  for  the  fusion  of  mission-related  data,  information  and  knowledge  in  the  context  of 
complex  missions  involving  co-operation  against  multiple  targets; 

(c)  Effective  deployment  of  crew  in  supervising  and  interacting  with  such  systems,  taking  into  account  cognitive  issues 
including  software  psychology  (addressing  human-software  interaction). 

TOPICS 

a)  Characteristics  of  operational  requirements  requiring  difficult  human-system  interaction,  e.g.  group  operations;  C^I 
issues;  interoperability;  uncertainty;  survival  in  battle  damage;  IFF:  consideration  of  air,  sea  and  land  systems. 

b)  Reliance  on  human  operators  to  perform  the  fusion  task  often  creates  a  prohibitively  high  workload  and  can  result  in  an 
unacceptable  reduction  in  the  timeliness  of  information.  This  applies  particularly  to  the  operation  of  modem  aircraft 
where  one  person  may  be  required  to  assimilate  all  the  sensor  information  as  well  as  flying  the  aircraft  and  managing 
other  systems.  Automating  the  functions  of  fusion  and  sensor  management  offers  a  potential  solution  to  these  workload 
and  timeliness  difficulties. 

c)  New  concepts  and  improving  techniques  in  the  integration  of  crew  with  complex  systems,  including:  solution  of 
cognitive  issues;  model-based  reasoning  for  human  advisory  systems;  the  psychology  of  human  interaction  with  complex 
software  systems;  virtual  reality;  de-cluttering  in  information  presentation;  the  multi-media  cockpit  including  head- 
mounted,  head-up  and  head-down  displays,  voice  input  and  output;  autonomous  machine  Vs  human-assisted  machine; 
interactions  and  transitions  between  system  autonomy  and  human  interaction;  situation  awareness;  representation  of  the 
human  operator  for  modelling. 


Theme 


Cette  reunion  a  pour  theme  la  collaboration  effective  entre  des  equipages  et  des  systemes  complexes  mais  non  totalement 

autonomes,  dans  un  contexte  de  sources  multiples  d’ informations  telles  que  communications  terrestres,  autres  plates-formes 

et  senseurs  multiples  sur  plates-formes.  Trois  elements  principaux  de  ce  theme  meritent  une  attention  particuliere  a  savoir  : 

(a)  les  Cciracteristiques  des  simations  operationnelles  impliquant  des  interactions  homme-systeme  difficiles 

(b)  la  conception  systemes  et  les  technologies  permettant  le  fusionnement  des  donnees  de  mission,  des  informations  et  des 
connaissances  dans  le  contexte  de  missions  complexes  requerant  des  actions  cooperatives  centre  des  objectifs,  multiples 

(c)  le  deploiement  efficace  des  equipages  en  ce  qui  conceme  la  surveillance  et  I’interaction  avec  de  tels  systemes,  compte 
tenu  des  questions  cognitives  y  compris  la  psychologie  de  I’interaction  homme-logiciel. 

SUJETS 

a)  caracteristiques  des  situations  operationnelles  exigeant  des  inter  actions  homme-systeme  difficiles  par  exemple  des 
operations  de  groupe;  questions  C3I;  interoperabilite;  incertitude;  survie  apres  avaries  au  combat  IFF;  la  prise  en  compte 
de  systemes  terre,  air  et  mer. 

b)  le  recours  a  des  operateurs  humains  pour  I’execution  de  la  tache  de  fusionnement  cree  souvent  une  charge  de  travail 
excessivement  lourde  et  risque  d’entramer  une  degradation  inacceptable  dans  I’actualisation  des  informations.  Ceci  vaut  a 
plus  forte  raison  pour  le  pilotage  de  1’ avion  de  combat  modeme  oil,  dans  certains  cas,  un  seui  homme  doit  assimiler 
I’enserable  des  informations  fourmes  par  les  senseurs,  piloter  I’avion  et  gerer  d’autres  systemes  en  meme  temps. 
L’automatisation  des  fonctions  de  fusionnement  et  de  gestion  des  senseurs  offre  une  solution  possible  a  ces  probl^mes  de 
charge  de  travail  et  d’actualisation. 

c)  I’application  de  concepts  nouveaux  et  1’ amelioration  des  techniques  d’integration  des  equipages  aux  systemes  complexes, 
y  compris  la  resolution  de  problemes  cognitifs;  le  raisonnement  a  partir  de  modeles  pour  des  systemes-conseil;  la 
psychologie  de  Finteraction  homme/logiciels  complexes;  la  realite  virtuelle;  I’elimination  sdlective  dans  la  presentation 
des  informations;  le  poste  de  pilotage  multimedia  y  compris  les  viseurs  de  casque  et  les  representations  tete  haute  et  tete 
basse,  les  entrees  et  les  sorties  vocales;  la  machine  autonome  centre  la  machine  assistee  par  Fhomme;  les  interactions  et 
transitions  entre  le  systeme  autonome  et  I’interaction  humaine;  I’appreciation  de  la  situation;  la  representation  de 
I’operateur  humain  pour  la  modelisation. 
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TECHNICAL  EVALUATION  REPORT 

Stanley  Leek 
8  Sunnyfield 
Hatfield 

HertfoidshireAL9  5DX 
United  Kingdom 


INTRODUCTION 

The  symposium  on  Sensor  Data  Fusion  and  Integration  of 
the  Human  Element  was  initiated  by  the  former  AGARD 
Mission  Systems  Panel.  The  theme  of  the  symposium  was 
stated  as  “the  effective  collaboration  between  crew  and  the 
complex,  but  not  entirely  autonomous,  systems  in  the  context 
of  multiple  information  sources  such  as  off-board 
communications,  other  platforms  and  multiple  sensors  on 
board  platforms.”  Particular  attention  was  given  during 
plaiming  to  consideration  of  human-system  interactions, 
system  design  and  technologies,  and  the  effective  deployment 
of  crew,  taking  into  account  cognitive  issues  and  psychology. 

The  main  topics  identified  for  the  symposium  can  be 
sununarised  as: 

■  The  characteristics  of  operational  requirements  requiring 
difficult  human-system  interaction  in  air,  sea  and  land 
systems  (e.g.  group  operations;  C?I  issues;  interoperability; 
uncertainty;  survival  in  battle  damage;  IFF). 

■  The  high  workload  and  information  delays  that  can  result 
from  reliance  on  human  operators  to  perform  the  fusion 
task,  particularly  in  relation  to  modem  aircraft  where 
multiple  sensor  inputs  may  be  loaded  on  to  a  single 
operator  in  addition  to  the  tasks  of  flying  and  system 
management. 

■  New  concepts  and  evolving  techniques  for  crew-system 
integraticMi,  embracing;  cognition,  model-based  reasoning, 
psychology,  virtual  reality,  uncluttered  information  present¬ 
ation,  the  multi-media  cockpit  (combined  head-up  and 
head-down  displays,  head-mounted  systems,  voice  input/ 
output  etc.),  autonomous  vs.  human-assisted  machines  plus 
interactions  and  transitions  between  the  two,  situation 
awareness,  and  human  operator  model  representation. 

Dr  John  Leggatt  of  the  Canadian  Department  of  National 
Defence,  gave  a  welcome  address  and  introduction  in  which 
he  outlined  the  activities  of  the  R&D  Branch  and  its  interest  in 
the  symposium.  He  gave  examples  of  relevant  Canadian 
activities  such  as  the  design,  development  and  evaluation  of  a 
computer-based  real  time  decision  support  system  (DSS) 
integrated  with  C^I  for  the  Modem  Frigate  programme,  and 
synthetic  displays  for  Search  and  Rescue  Helicopters  (the 
latter  being  one  of  the  highlights  of  a  Technical  Tour  of  the 
Defence  Research  Establishment  Ottawa  (DREO)  which  took 
place  during  the  week). 

KEYNOTE  ADDRESS 

In  his  Keynote  Address,  “Sensor  Data  Fusion  Technologies 
for  Precision  Strike,”  Eugene  L  Freeman  of  the  Boeing 
Company,  US,  gave  an  assessment  of  the  enabling  tech¬ 
nologies  for  sensor  data  fusion  in  the  context  of  precision 
strike  for  a  2010  time  frame.  He  emphasised  a  view  of  data 
fusion  as  a  means  for  adding  leverage  to  the  human  element 
rather  than  replacing  it.  Precision  strike  operations  make  big 


demands  on  the  operator  and,  at  each  stage  of  an  operation, 
involve  a  variety  of  operator  actions  for  which  sensor  fusion 
can  provide  major  benefits.  He  went  on  to  discuss  sensor 
design  considerations  in  relation  to  various  military 
applications,  atmospheric  and  weather  effects,  the  enabling 
effect  of  micro-electro-mechanical  systems  (MEMS),  process¬ 
ing  capacity  and  cost  improvements,  and  GPS  integration; 
expected  target  sensor  attributes  for  a  range  of  sensor  types, 
in  terms  of  weather,  clutter,  automatic  target  recognition 
(ATR)  discriminants  and  design,  weight,  and  cost;  and  sensor 
data  fusion  architecture  with  fusioi  at  different  levels  from 
the  decision-level  downwards,  including  their  relative  merits, 
and  the  ongoing  debate  regarding  sensor-level  versus  pixel- 
level  fusion  in  the  sensor  data  fusion  community. 

He  concluded  by  stressing  the  importance  of  objective 
assessment  of  alternative  systems  and  described  an  evaluation 
study  which  tries  to  balance  the  system  level  “requirements 
pull”  with  the  sensor  and  ATR  communities’  “technology 
push.” 

SESSION  I:  CHARACTERISTICS  OF  OPERATIONAL 
REQUIREMENTS 

Two  of  the  papers  originally  planned  for  this  session  had 
been  withdrawn.  Mr  Jean  Bernard  Senneville,  FR,  introduced 
the  session's  remaining  two  papers  as  snapshots  of  system 
concepts  driven  by  requirements. 

The  first  paper  of  the  session,  “Intelligent  Crew  Assistant 
for  Military  Transport  Aircraft”,  by  Walsdorf  and  Orrken  of 
the  University  of  the  Armed  Forces,  GE,  was  presented  by 
Anton  Walsdorf.  It  described  the  requirements  for  CAMA,  the 
Crew  Assistant  Military  Aircraft,  and  its  design  principles. 
Recent  accident  investigations  were  stated  as  evidence  of 
aircrew  being  overtaxed  by  current  concepts  of  cockpit 
automation.  Automation  -  which  can  lead  to  increased  aircraft 
performance  and  extend  mission  capabilities  but  still  needs 
crew  monitoring  and  supervision  -  can  make  the  multitask 
environment  of  piloting,  navigating,  commrmicating  and 
systems  management  more  difficult  in  demanding 
circumstances.  A  video  film  illustrated  the  difficulties  involv¬ 
ed  in  a  military  transport,  with  minimal  ground  control 
assistance,  making  a  steep  approach  over  a  hiU  to  a  landing 
strip  or  a  drop  zone  in  order  to  avoid  air  defences. 

The  paper  identified  the  need  for  a  system  to  assist  the  pilot 
in  the  cognitive  process  and  in  decision  taking,  and  went  on  to 
analyse  the  characteristics  of  a  cognitive  assistant  system  in 
which  conventional  automated  features  are  augmented  by 
knowledge-based  or  rule-based  functions  of  recognition, 
identification  and  decision  making,  and  overlap  with  more 
conventional  automation  of  plarming,  task/state  association, 
and  feature  formatirm  from  sensory  inputs.  The  CAMA  design 
was  described  in  terms  of  interpretation  of  the  situation, 
interpretation  of  pilot  behaviour,  situation  diagnosis,  mission 
planning,  and  crew  interface.  The  paper  concluded  with  a 
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brief  statement  of  flight  simulator  trials  carried  out  in  1997/98 
and  in-flight  experiments  planned  for  early  2000  (the  co¬ 
author,  Reiner  Onken.  pointed  out  that  further  description  of 
simulator  trials  were  contained  in  a  paper  to  be  delivered  in 
Session  III).  In  answer  to  a  question  from  the  Session 
Chairman  regarding  the  effects  of  human  error,  the  author 
stated  that  crew  responses  were  continually  reviewed  in 
closed  loop  simulations  and  said  that  the  system  had  never 
failed  to  recognise  pilot  intent. 

The  other  paper  in  this  session,  “On  the  Design  of  a 
Decision  Support  System  for  Data  Fusion  and  Resource 
Management  in  a  Modem  Frigate",  by  Bmce  A  Chalmers  of 
the  Defence  Research  Establishment  Valcartier,  CA,  was 
concerned  with  mid-life  upgrade  of  the  Command  and 
Control  system  (CCS)  for  the  HALIFAX  Qass  frigate  in  the 
2005-2010  timeframe.  The  paper,  though  including  many  of 
the  same  cognitive  issues  as  the  first  paper,  was  more 
concerned  with  the  underlying  theoretical  basis  for  design  of 
the  human-system  interface  and  the  different  perspective  of 
maritime  operations  in  open-ocean  and  littoral  environments 
(the  latter  presenting  particular  difficulties  of  geographical 
CMistraints,  increased  numbers  of  threats,  sensor  and  guidance 
systems  degradation,  and  commercial  air  and  surface  traffic 
congestion). 

The  author  reviewed  the  characteristics  of  shipboard  C2, 
describing  the  Combat  System  Operator  (CSO)  team  in  the 
ship's  Operations  Room,  oiganised  hierarchically  with  sub¬ 
division  generally  into  warfare  areas  such  as  air,  surface,  and 
sub-surface.  He  provided  a  background  philosophical  analysis 
of  the  issues  involved,  including  problems  and  opportunities 
related  to  goals,  and  a  view  of  C2  as  a  complex  sociotechnical 
system  involving  a  wide  range  of  operational  problems  and 
teamwork.  He  also  noted  various  navy  research  efforts  in  the 
USA  and  UK  directed  towards  leaner  crewing  of  future  ships. 
He  discussed  the  automation  issues  of  decision  aid  metaphors 
(as  either  prostheses  or  tools)  and  dynamic  delegation  of 
authority  (with  alternative  modes  appropriate  to  changing 
circumstances),  before  describing  a  cognitive  model  ofC2  and 
the  approach  to  a  framework  for  design  based  on  an 
abstraction-decomposition  hierarchy  from  the  top  level 
functional  purpose  of  the  whole  system  down  to  the  physical 
form  of  individual  components,  and  finally  a  discussion  of 
design  goals.  The  work  is  expected  to  lead  to  a  specification 
of  a  decision  support  system  integrated  with  existing  CCS  to 
support  operators  with  data  fusion,  situation  and  threat 
assessment,  and  resource  management.  He  confirmed,  in 
answer  to  a  question  that  the  work  had  not  yet  preceded 
beyond  the  exploration  of  concepts  and  development  of 
algorithms. 

The  two  papers  in  this  session  provided  striking  contrasts  in 
system  characteristics  for  maritime  warfare  and  air  transport 
scenarios.  The  organic  command  and  control  system  of  a 
frigate  and  its  crew  needed  to  handle  a  wide  variety  of 
missions  and  tasks  is  very  different  from  the  single-mission 
but  complex  task  of  the  individual  pilot  of  a  military  transport 
aircraft.  Nevertheless  the  similarities  that  emerged,  in  regard 
to  the  requirements  for  cognitive  and  decision  aids  for 
operators  faced  with  the  growing  magnitude  of  systems  and 
sensor  data,  provided  an  instractive  introduction  and 
background  to  the  rest  of  the  symposium. 

SESSION  II:  SYSTEM  DESIGN  TECHNIQUES  AND 
TECHNOLOGIES 

The  six  papers  of  this  session,  which  was  introduced  by 
Ir  Pieter  van  den  Broek,  NE,  who  was  also  Chairman  of  the 


symposium,  were  concerned  with  concepts  for  data  fusion  in  a 
variety  of  applications. 

The  first  paper,  introduced  by  Ma±  Alford  and  presented 
by  Dale  Klamer,  was  on  “Measures  of  Merit  for  Collaborative 
Collection,  Connection,  and  Execution  Management”  by 
Kowalski,  Stubberad  and  Klamer  of  ORINCON  Corporation 
and  Alford  of  the  Air  Force  Research  Laboratory,  US.  The 
paper  described  a  future  technique  for  information  sharing 
between  otiierwise  independent  sensor  platforms,  using 
interactive  measures  of  merit  (MOMs).  The  top-level  design 
scheme  to  be  used  in  individual  platforms  would  use  MOMs 
in  data  collecticsi  and  execution  of  the  fusion  process,  in 
conjunction  with  a  connection  management  process  utilising  a 
connection  agent  to  execute  infomiation  requests  and 
respcmses  to  and  from  other  platforms.  The  scheme  aims  to 
provide  an  interactive  envircmment  for  distributed  intelligence 
agents  (e.g.  individual  platforms/sensors). 

A  key  feature  of  the  scheme  is  the  use  of  metrics  to 
determine  the  quality  of  the  collection  plan  generated  in  the 
collection  process,  the  quality  of  gathered  information  and  the 
quality  of  connections.  The  approach  offers  the  potential  for 
efficient  theatre-wide  surveillance,  targeting  and  tracking  by  a 
two-way  collaborative  process  between  multiple  platforms 
and  sensors.  System  design  and  development  has  reached  the 
stage  of  individual  component  testing.  In  answer  to  a  question 
on  the  benefits  of  dynamic  evaluation  of  MOMs,  the  authors 
mentioned  tire  possibility  of  revising  parameters  to  allow 
dynamic  reconfiguration  and  its  importance  in  a  changing 
battlefield. 

The  next  paper,  “Contextual  Information  and  Multisensor 
Data  Fusion  for  Battlefield  Applications”  by  Bastifere, 
Chalmeton  and  Nimier,  FR,  focused  on  algorithms  for  the 
more  specific  task  of  multi-sensor  tracking  of  individual 
targets.  It  was  presented  by  Aimie  Bastifeie  who  gave  a 
detailed  exposition  of  the  principles  involved.  Extended 
Kalman  filters  are  applied  independently  to  each  of  the  sensor 
measurement  coordinates,  rather  titan  to  the  whole  cartesian 
representation  of  the  state  variables.  This  approach  was  aimed 
at  avoiding  global  deterioration  of  the  results  in  the  event  of 
individual  coordinate  degradation.  As  well  as  robustness  to 
perturbatiois,  other  advantages  claimed  for  this  approach 
included  simpler  (linear)  filters  and  no  need  for  inversion 
matrices.  The  paper  presented  comparative  mathematical 
analyses  of  a  classical  multisensor  tracking  method  and  a 
multisensor  tracking  method  taking  context  into  account  (with 
contextual  parameters  enabling  sensor  validity  ranges  to  be 
characterised  with  membership  functions  relative  to  fuzzy 
subsets). 

Numerical  simulations  have  confirmed  the  superiority  of 
the  context-sensitive  approach.  The  possibilty  of  including 
further  contextual  infoimation,  such  as  weather  and  extending 
the  techniques  to  other  scenarios  (e.g.  multiple  targets),  was 
mentioned.  The  author  confirmed,  in  answer  to  questions,  the 
possibility  of  extending  the  scope  of  contextual  information 
from  other  sources  and  noted  the  principal  application  at 
present  to  relatively  slow-moving  targets  such  as  AFVs. 

The  paper  “Image  Data  Fusion  for  Future  Enhanced  Vision 
Systems”  by  Drihler,  Hecker  and  Rodloff  of  the  Institute  of 
Flight  Guidance,  DLR,  GE,  was  presented  by  Dr  Hans-Ulrich 
Dohler.  It  included  comparisons  of  competing  imaging 
sensors  in  the  context  of  the  needs  of  military  transport 
aircraft,  as  well  as  the  results  of  recent  field  trials,  and  first 
concepts  for  the  fusirai  of  radar  and  synthetic  images  -  the 
basis  for  the  erthanced  vision  system  decribed.  The 
advantages  and  disadvantages  of  direct  sensor  vision  and 
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synthetic  vision  were  reviewed;  in  particular  the  difficulties  of 
image  interpretation  with  direct  sensor  displays  and  the 
problems  of  obstacle  detection,  reliability  and  accuracy  with 
synthetically  generated  displays.  The  aim  of  the  enhanced 
vision  system  was  to  make  use  of  all  the  information  available 
in  combination  with  on-board  aircraft  stams  and  ATC  inputs. 

The  paper  reviewed  the  characteristics  of  sensor  types 
ranging  from  UV  video  and  LADAR,  through  medium  and 
long  waveband  IR,  to  active  and  passive  MMW  radar  bands 
up  to  35GHz;  from  which  it  was  concluded  (not  surprisingly) 
that  for  military  trarrsport  aircraft  operating  in  in  poor  weather 
conditions,  active  MMW  radar  would  provide  the  best  all- 
rotmd  solution.  Trials  were  conducted  using  a  MMW  FMCW 
radar  of  high  sensitivity  and  low  detectivity  (important  for 
military  applications)  mounted  on  top  of  a  DO-228  twin 
turboprop  aircraft.  A  technique  of  “data  fusion  in  time  and 
space,”  using  sensor  images  at  different  times  as  well  as  from 
different  sources  was  used  to  generate  scene  images  in  a 
variety  of  presentation  modes,  with  promising  results.  The 
questions  that  followed  were  mainly  concerned  with  clutter 
discrimination  and  update  rates;  for  which  the  author  cited 
acceptable  experimental  results,  though  (in  answer  to  a 
question  by  the  Chairman)  the  system's  usefiiiness  during 
taxying  would  require  further  trials. 

The  fourth  paper  in  Session  II,  “A  distributed  System  for 
Command  and  Control  Applications  with  Programming 
Language  Abstraction”  was  given  by  the  author,  Ethan 
Satidogan  of  the  Turkish  Navy’s  Software  Development 
Centre.  Similar  in  its  application  to  the  paper  by  Chalmers  in 
Session  I,  this  paper  focused  on  software  techniques 
(particularly  the  use  of  object  oriented  programming)  rather 
than  the  overall  system.  The  author  reviewed  the  character¬ 
istics  and  requirements  of  distributed  and  real-time  computing 
including  OOP  (object  oriented  programming)  languages, 
concurrency,  process  management,  communication,  inherit¬ 
ance  and  real-time  aspects,  before  describing  CPL  (the 
CORD  Programming  Language).  This  is  based  on  C+H-  with 
new  keywords  and  a  special  layer  to  handle  distributed 
processors.He  also  described  the  CORD-RTS  (Concurrent, 
Object  oriented  and  Real-time  Distribution  Run-Time 
System).  Performance  details  were  given,  in  terms  of  inter¬ 
object  communication  times  achieved  for  three  different 
computer  platforms.  The  command  and  ccaitrol  system 
context  and  its  links  were  mentioned  as  an  introducticHi  to  a 
description  of  a  proposed  solution  to  applications, 
covering  hardware  and  software  architecture,  and 
implementaticHi  -  with  special  attention  to  track  management, 
using  either  a  centralized  model  or  a  distributed  model  for 
sensor  fusion.  In  his  concluding  remarks,  the  author  reiterated 
the  value  of  linguistic  mechanisms  for  abstracting  real-time 
constraints  and  distribution,  including  simplified  program¬ 
ming.  The  work  is  still  under  development  at  the  TNSDC.  The 
author,  when  asked  how  well  CST  compared  with  Java,  was 
confident  in  its  superiority  for  the  given  application. 

The  paper  “Novel  Concepts  for  Identity  Fusion  in  an  Air 
Combat  Enviromnent,”  was  presented  by  Shari  Atman  of 
DERA,  UK,  as  an  AI  approach  to  augmenting  current  data 
fusion  algorithms  for  air  combat  applications.  Fused  sensor 
outputs  of  target  tracks,  generated  through  extended  Kalman 
filters,  produce  software  objects  which  are  further  pro¬ 
grammed  with  a  set  of  self  beliefs  including  variable 
behavioural  response,  goals  and  knowledge  cH  how  to  achieve 
those  goals.  The  author  described  the  programme  of  ongoing 
work,  including  the  Sensor  Tracking  Application  Rig  (STAR), 
incorporating  HOTAS  (hands  on  throttle  and  stick)  man-in- 


the-loop  interface,  before  discussing  track  identity  fusion  and 
the  potential  -  and  concepts  -  for  augmentation,  and  intro¬ 
ducing  the  concept  of  intelligent  agents  as  meta-objects 
embodying  cognitive  notions  of  beliefs,  desires  and 
intentions.  The  information  agent  structure  of  self  model, 
acquaintance  model,  world  model  and  ability  model  form  the 
basis  for  an  optimised  system  architecmre  with  an  agent 
wrapper  built  around  a  multi-sensor  data  fusion  tracking 
algorithm.  Asked  if  it  might  not  be  better  to  have  separate 
agents  for,  say,  identification  and  identity,  the  author  replied 
that  this  would  lead  to  a  high  degree  of  dupUcation,  when 
compared  to  the  proven  ability  to  combine  them  in  a  single 
agent  wrapper. 

The  final  paper  in  Session  II,  “MIDS  (Multi-functional 
Information  Distribution  System)  Triangulation,”  by  Ziegler 
and  Sachsenhauser  of  Daimler-Benz  Aerospace,  GE,  was 
presented  by  Joseph  Ziegler.  The  system’s  aim  is  to  improve 
the  range  spoke  of  a  platform’s  sensor  (especially  against 
jammers)  by  reference  to  information  on  target  range  or  track 
from  other  platforms.  The  triangulation  and  de-ghosting 
process  described  assumed  direction  spokes  from  MIDS  (a 
NATO  system  to  provide  for  data  exchange  among  airborne, 
land  and  sea  forces).  An  exposition  of  the  triangulation  math¬ 
ematics  was  given,  including  extrapolation  of  ownship  data, 
derivation  of  the  intersection  equations  (of  spokes  from  the 
platform's  own  sensor  and  from  MEDS)  and  alignment.  Ind¬ 
ividual  analyses  were  given  of  the  credibility  of  MIDS  range, 
visibility,  height,  vertical  velocity,  manoeuvre  and  velocity, 
leading  to  the  credibility  of  the  intersection  point  itself,  and 
alternative  association  tests  were  described. 

Results  were  shown  of  simulations  of  the  triangulation 
algorithms  (using  a  PC),  carried  out  mainly  for  the  purpose  of 
fine  tuning  constants.  It  was  interesting  to  note  that  88%  of 
processing  time  was  taken  up  by  the  algorithm,  transform¬ 
ations,  and  range  and  range  rate  calculation,  and  only  12%  by 
credibility  and  association  tests.  It  was  confirmed,  in  answer 
to  questions,  that  time  histories  were  not  determined  (or  used), 
only  variances;  and,  despite  the  notoriously  poor  accuracy  of 
aircraft  ESM  sensors,  useful  results  had  been  achieved. 

SESSION  III:  INTEGRATION  OF  HUMAN  OPERAT¬ 
ORS  WITH  COMPLEX  SYSTEMS 

This  session,  chaired  by  Dr-Ing  Luigi  Crovella,  IT,  was 
originally  planned  to  include  five  papers,  one  of  which  was 
moved  to  Session  IV. 

The  first  paper  of  the  session,  “Subjective  Information 
Given  by  Users  in  Bearings-only  Target  Motion  Analysis,”  by 
Spigai  and  Grandin  of  Thomson-CSF-RCM,  FR,  was  present¬ 
ed  by  Jean-Fran^ois  Grandin.  It  addressed  a  similar  problem 
to  that  of  the  final  paper  of  the  previous  sessicai  -  determining 
the  position  and  velocity  of  a  target  from  sensors  in  more  than 
one  location  -  but  with  more  limited  information,  and 
including  a  subjective  human  element  In  the  Bearings-only 
Target  Motion  Analysis  (BTMA)  scheme  described,  two 
observers  on  the  ground  measure  bearings  of  manoeuvring  air 
targets  which  are  used  in  a  trajectory  estimaticHi  algorithm 
based  on  Hidden  Markov  Models  that  allow  the  use  of 
subjective  information.  The  paper  contained  a  review  of  the 
limitations  of  the  extended  Kalman  filter  for  non-linear  or 
non-Gaussian  problems  and  the  alternative  interacting 
multiple  models  (IMM)  based  on  multiple  extended  Kalman 
filters.  There  followed  a  statement  of  the  principles  of  the 
hidden  Markov  models  (HMM).  Subjective  information  from 
the  human  observer,  typically  in  the  form  of  estimates  of 
possible  target  position,  is  input  to  a  map  display  which  also 
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provides  additional  aids  such  as  geodesic  information,  type  of 
target,  etc.  A  fairly  simple  algorithms  is  employed  to  provide  a 
best  estimate  of  target  position.  Simulation  has  demonstrated 
smooth  and  accurate  results  for  HMM  (subject  to  the  quality 
of  the  subjective  inputs)  when  compared  to  the  more 
mechanistic  IMM.  As  a  next  step,  it  is  intended  to  develop  an 
automatic  learning  process  to  aid  the  user. 

The  next  paper,  “Fusion  and  Display  of  Data  According  to 
the  Design  Philosophy  of  Intuitive  Use,”  was  by  Goetz  Ardey 
of  the  Technical  University  of  Braunschweig,  GE.  His 
historical  survey  showed  how  the  ad  hoc  development  of 
avionics  systems  had  led  to  severe  demands  on  aircrew,  with 
worrying  implications  for  safety,  particularly  in  the  field  of 
General  Aviation.  Examples  of  past  and  present  cockpits 
illustrated  the  growth  in  the  number  of  displays  and  controls 
confrcsiting  pilots,  (who  may  be  of  widely  varying  back¬ 
grounds  and  abilities).  In  addition,  he  pointed  out  that 
equipments  developed  for  large  commercial  aircraft  were 
generally  unsuitable  (too  big,  too  complex,  too  expensive)  for 
GA  aircraft.  The  author  went  on  to  describe  the  COSIMA 
programme.  COSIMA  (Cockpit  and  Simulator  for  General 
Aviation)  aims  to  integrate  new  technologies  to  provide  an 
intuitive  user  interface  based  on  operational  analysis  of 
cockpits.  The  system,  installed  in  a  Cessna  C-172  testbed, 
consists  of  two  PC-like  computers  with  inputs  from  sensors 
(engine  instruments,  GPS,  etc.)  and  controls  (keyboard, 
trackball)  and  outputs  to  three  side-by-side  colour-LCD 
screens.  The  centre  display  shows  the  Aircraft  Monitoring 
System,  AMS  (essential  engine,  fuel,  and  time  data)  with 
optional  levels  of  information  controlled  by  keyboard  input. 
The  side-displays  show  Flight  Planning  and  Navigation 
System  (FPNS)  information  in  the  form  of  moving  maps, 
together  with  indicated  air  speed,  altitude  (and  rates  of 
change)  heading,  ground  track,  etc.  The  cockpit  layout  aims  to 
achieve  a  well-balanced  presentation  with  variable  operational 
modes  and  minimal  distraction  from  the  flying  task.  On  a 
question  of  pilot  acceptability,  the  response  of  those  who  had 
tried  it  was  said  to  be,  “when  can  we  have  it?”  Other 
questions  were  concerned  with  IFR  capability  (not  practical  in 
its  present  form)  and  multiple  AMS  fault  display  (handled  by 
prioritization  of  the  different  levels). 

The  paper  “Enhanced  and  Synthetic  Vision  System  Con¬ 
cept  for  Application  to  Search  and  Rescue  Missions,”  by 
Swail  and  Jennings  of  the  Institute  for  Aerospace  Research, 
CA,  was  presented  by  Carl  Swail.  Search  and  Rescue  by 
helicopters  is  an  important  issue  for  Canada,  given  its  size  and 
need  to  access  remote  areas,  often  in  poor  weather  (down  to 
100  ft  ceiling  and  1/8  nm  visibility).  The  system  described  by 
the  author  is  intended  to  augment  pilot's  situational  awareness 
in  four  phases:  descent  in  a  search  area  without  external  aids; 
search  for  crash  site,  survivors  and  wreckage;  transition  to, 
and  maintenance  of  hover  during  rescue;  and  safe  egress  from 
the  rescue  area.  Constmcted  largely  from  COTS  components, 
it  includes  a  camera  platform  with  infrared  and  visual  band 
sensors,  an  image  generation  system  incorporating  a  terrain 
database  and  with  inputs  from  a  range  sensor,  and  a  digital 
image  processing  unit  which  fuses  video  and  synthetic  images 
to  provide  an  output  to  the  pilot’s  helmet-mounted  display, 
plus  a  head  tracker  which  controls  the  camera  platform,  HMD 
display  and  the  DIPU. 

A  prototype  system  is  due  to  be  incorporated  in  a  Bell  205 
by  2000:  whilst  an  ongoing  research  programme  has  been 
addressing  the  questions  of  field-of-view,  camera  platform  roll 
compensation,  system  delays,  display  symbology,  image 
fusion  techniques,  tme  binocular  vs  bi-ocular  trades,  and 


synthetic  image  scene  content.  In  answer  to  a  question,  the 
author  stated  that  a  range  sensor  was  not  being  incorporated 
initially  but  eventually  an  infrared  sensor  was  envisaged. 

The  final  paper  in  Session  III,  “Evaluation  of  the  Cockpit 
Assistant  Military  Aircraft  CAMA  in  Simulator  Trials,”  by 
Schulte  (ESG  Elektroksystem-  und  Logistik-GmbH)  and  Stiitz 
(Universitat  der  Bundeswehr),  GE,  was  presented  by  Peter 
Stiitz.  The  CAMA  system  is  that  described  in  the  paper  by 
Walsdorf  and  Onken  in  Session  I.  The  flight  simulator  has 
primary  flight  displays  either  as  standard  ADI  (for  IFR 
phases)  or  3-D  perspective  format  (for  low  level  flight),  plus  a 
multi-mode  interactive  navigation  display  with  a  touch- 
sensitive  screen  input  for  mode  selection,  etc.  Speech 
recognition/synttiesis  is  used  in  parallel  with  the  manual/ 
visual  input/output. 

Ten  German  Air  Force  transport  pilots  took  part  in  the 
trials,  with  various  scenarios  and  tasks,  between  Autumn 
1997  and  Spring  1998.  Subjective  ratings  were  given  by  the 
pilots  in  terms  of  evaluations  of  flight  simulation,  CAMA 
philosophy,  situation  awareness,  assistance  quality,  and 
advice  and  warning  philosophy.  The  pilots  were  also  asked  to 
rate  the  handling  qualities  and  acceptance  of  CAMA.  Finally 
they  were  asked  to  provide  a  general  evaluation,  with  ratings 
of  CAMA’s  influence  on  flight  safety,  mission  efficiency,  and 
survival  probability.  Overall,  the  system  was  rated  as  “very 
good”.  Objective  evaluations  in  the  form  of  reaction  times  in 
response  to  events  such  as  appearance  of  a  threat,  change  of 
destination,  and  corridor  change,  were  given,  showing  a 
dramatic  improvement  -  the  result,  it  was  stated,  of  reduction 
in  mental  workload.  The  trials  were  also  effective  in 
indicating  improvements  in  display  formats.  Further  work 
plaimed  includes:  flight  trials  (scheduled  for  2000)  and 
integration  of  imaging  sensor  information  into  the  3-D  flight 
guidance  display  to  achieve  an  enhanced  vision  system.  In 
answer  to  questions,  the  author  said  that  objective  tests  to 
back  up  aU  of  the  subjective  evaluations  were  not  possible 
(for  example,  situation  awareness).  He  conceded  that  the  use 
of  operational  pilots  may  not  be  ideal  when  compared  with 
professional  test  pilots’  critical  appraisal  of  new  technology, 
but  had  been  the  only  practical  trials  approach. 

SESSION  IV:  SYSTEM  APPLICATIONS 

This  session,  chaired  by  Mr  Paul  Seigent,  FR,  contained 
five  papers,  including  one  (the  last  paper)  transferred  from 
Session  III. 

The  first  paper  of  the  session,  “Fusion  and  Display  of 
Tactical  Information  Within  Battlefield  Helicopters,”  by  Watts 
and  Silvester  of  DERA,  UK.  Presented  by  Arma  Watts;  its 
subject  was  the  trials  by  the  Defence  Evaluation  and  Research 
Agency  of  alternative  forms  of  Tactical  Situation  Displays 
(TSD)  for  the  WAH-64  attack  helicopter.  The  author  described 
the  HOVERS  man-in-the-loop  mission  simulator,  the  cockpit 
configuration  (appropriate  to  a  possible  2010  timeframe  for 
introducing  TSD),  TSD  configurations,  trial  scenarios  and 
threat  environment  (defined  by  the  Army  Air  Corps),  and 
trials  procedure  (with  72  AAC-crewed  missions).  Results 
were  shown  for  three  TSD  configurations.  The  “baseline”  case 
assumed  on-board  sensor  data  available  at  the  in-service  date 
overlaid  on  a  digitised  map  containing  tactical  information. 
The  “improved”  and  “fused”  cases  have  different  degrees  of 
integration  of  on-board  sensor  and  tactical  information. 

The  trials  results  were  illuminating.  Although  the  probab¬ 
ility  of  being  killed  by  ADUs  reduced  dramatically  and 
progressively  with  the  improved  and  fused  cases,  overall 
survival  was  only  slightly  improved,  due  to  an  increase  in  the 
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probability  of  being  killed  by  threat  helicopters.  This  result 
was  ascribed  to  commanders  focusing  their  attention  on  the 
map  displays  -  used  successfully  to  improve  use  of  terrain  to 
avoid  ADUs  -  to  the  detriment  of  survival  against  threat 
helicopter  missiles.  It  was  nevertheless  concluded  that  the 
simpler  tactical  display  made  possible  by  fusion  of  on-board 
sensor  data  improved  crew  planning  and  survivability.  It  was 
also  concluded  that:  comprehensive  training  will  be  needed  to 
realise  the  full  benefits;  appropriate  enhanced  information 
should  be  provided  to  both  the  pilot  and  the  commander  to 
improve  communication;  and  further  benefits  would  be  gained 
from  on-board  planning  capability  and  additional  display 
functiaiality.  Questioned  on  timeliness  and  validity  of  the 
data  displayed,  the  author  replied  that  human  factors  experts 
had  been  consulted  regarding  display  of  threat  error  circles 
but  their  size  would  render  them  of  doubtful  utility. 

The  second  paper  in  the  session,  by  Roschmann  and 
Wacker  of  Daimler-Benz  Aerospace,  GE,  was  on  “The  Multi- 
Sensor  Integration  System  for  NATO  E-3A  Mid-Term 
Modernisation”  and  presented  by  Ulrich  Pietzschmann.  The 
paper  described  an  integrated  programme  of  work  embracing 
hardware  and  software,  starting  with  the  historical 
background,  and  a  programmme  overview.  The  E-3A  mid¬ 
term  modemisaticai  programme  reflects  the  changed  -  and 
more  fragmented  -  geo-political  situation  that  has  arisen  since 
NATO  AWACS  was  first  introduced.  This  makes  greater 
demands  on  the  system,  in  terms  of  quickly-changing 
situations  and  an  increased  number  and  variety  of  inputs.  It 
will  introduce  new  equipments  for  communications, 
navigation  (including  GPS)  and  identification,  as  well  as  the 
integration  of  all  available  sensor  data,  state-of-the-art  flat 
panel  displays,  and  new  COTS-based  computers  with  ojjen 
systems  architecture  and  future  upgrade  capacity. 

The  MSI  system  is  required  to  provide  a  unique  display 
object  from  all  the  inputs  related  to  each  single  target  Its 
software,  coded  in  Ada95  and  C/C-H-,  is  executable  on  a 
SUN/SPARC  compatible  computer  with  Solaris  2.x  OS,  and  is 
required  to  communiGate  wiA  the  AWACS  mission  control 
(AMC)  computer  via  a  Gigabit  Ethernet  Outline  descriptions 
of  the  multi-sensor  tracking  and  identification  systems,  and 
the  MSI  manager,  were  followed  by  a  review  of  the  test 
philosophy  which  utilizes  a  DAS  A  MSI  Software  Test  Bed 
which  permits  testing  independently  of  the  Boeing  AMC.  In 
answer  to  questions  on  the  approach  to  identification  and  data 
fusion,  it  was  stated  that  a  rule-based,  rather  than  probabilistic 
approach  was  followed,  with  updates  based  on  operator 
indications.  Although  it  was  agre^  tiiat  this  does  not  follow 
current  STANAGs,  it  was  nevertheless  accepted  by  the  user. 

The  next  paper,  “Environmental  Perception  Process  in 
Maritime  Command  and  Control,”  by  Paradis  and  Roy  of  the 
Defence  Research  Establishment,  Valcartier,  CA,  and 
Treumiet  of  TNO,  the  Hague,  NE,  was  presented  by  Stdphane 
Paradis.  It  was  complementary  to  the  paper  by  Chalmers  and 
Bosse  in  Session  I,  being  concerned  with  the  same  subject 
matter  of  maritime  C?  but  concentrating  on  situaticm  aware¬ 
ness  and  data  fusion.  The  environment  perception  process 
(EPPj  was  presented  as  a  step  towards  defining  an  integrated 
architecture  for  data  fusion  and  a  basis  for  higher  levels  of 
abstraction.  A  conceptual  model  provided  by  Boyd’s  Observe- 
Orient-Decide-Act  (OODA)  loop  was  given  as  a  means  of 
capturing  the  command  and  control  process,  onto  which  could 
be  mapped  the  elements  of  multi-sensor  data  fusion,  situation 
and  threat  assessment,  plan  creation,  and  plan 
implementation.  This  provided  the  background  for  a  discuss¬ 
ion  of  a  situation  awareness  perspective  for  data  fusion  and 


an  exposition  of  the  environment  perception  process  (EPP) 
itself,  representing  the  first  two  levels  of  Endsley’s  three-level 
model  of  situation  awareness  (perception,  comprehension,  and 
projection).  An  EPP  framework  was  presented,  supported  by 
examples,  the  first  of  which  illustrated  a  situation  in  which 
three  targets  ate  tracked,  whilst  a  third  target  track  -  inferred 
from  a  knowledge  of  doctrine  -  may  be  constructed,  with 
inferences  for  the  total  situation.  The  author  concluded  with  a 
discussion  of  uncertainty  and  potential  pitfalls  -  for  example, 
the  danger  of  data  looping  where  information  from  higher 
data  fusion  levels  may  be  fed  back  into  the  fusion  process  as 
if  it  were  independent  information.  Questions  were  mainly 
concerned  with  the  issue  of  identity  uncertainty  raised  in  the 
final  part  of  the  paper,  the  author  replying  that  it  was  not  clear 
whether  this  should  be  output.  The  Session  Chairman 
commented  that  too  much  information  can  make  decision 
making  more  difficult,  a  problem  that  an  integrated  data 
fusion  system  ought  to  be  trying  to  avoid. 

The  last  of  the  papers  originally  scheduled  for  this  session, 
“Tactical  Missions  of  Transport  Aircraft:  an  Approved  Low 
Level  Guidance  Concept  Reduces  Crew  Workload,”  by 
Lerche,  Schoder  and  Mehler  of  Daimler-Benz  Aerospace,  was 
given  by  Dipl-Ing  H-D  Lerche.  The  difficulty  of  flying 
transport  aircraft  at  low  altitude  for  long  periods  (up  to 
SOOnm),  led  to  the  system  concept  described  in  this  paper.  The 
author  reviewed  the  tactical  and  delivery  reasons  for  low-level 
flying  and  the  problem  of  terrain  avoidance  coupled  with 
freedom  of  spontaneous  lateral  manoeuvre,  which  demand  an 
exceptional  degree  of  concentration  on  the  part  of  the  pilot. 
The  Low  Level  Flight  Guidance  Demonstrator  concept  design 
is  based  on  a  rolling  stone  algorithm  in  which  flight  path 
radius  is  overlaid  on  the  terrain  profile  generated  from  a 
digital  data  base,  to  generate  a  flyable  altitude  profile.  Energy 
management  is  also  included  in  the  algorithm. 

Experimental  equipment  for  flight  demonstration  and  valid¬ 
ation,  consisting  of  a  guidance  computer,  HUD  and  HDD,  has 
been  implemented  in  the  ATTAS  (Advanced  Technology  and 
Testing  Aircraft  System)  with  the  aim  of  demonstrating  pilot’s 
increased  situation  awareness,  optimizing  the  man/machine 
interface,  and  testing  handling  qualities.  The  pilot  interface 
emphasises  care  free  moding  with  pilot  selection  between 
fixed  track  (fully  automatic  3-D  pre-planned  track  with  auto¬ 
throttle),  free  track  (automatic  altimde  and  throttle  with  pilot 
lateral  control),  and  standby  track  (pilot  steering  with 
synthetic  vision  and  ground  proximity  warning).  Support  to 
the  programme  is  also  provided  by  the  parallel  Reliable  Auto¬ 
nomous  Precise  Integrated  Navigation  (RAPIN)  C-160 
Transall  flying  testbed  -  which  has  demonstrated  data  fusion 
of  navigation  sensors  LINS,  GPS  and  radar  altimeter/TRN.  In 
reply  to  questions,  the  author  stated  that  the  display  was 
selected  on  the  basis  of  pilots’  preferences  from  a  study  by 
Darmstadt  University.  This  woik  confirmed  the  differences  in 
display  options  -  and  needs  -  compared  to  Tornado. 

The  final  paper  in  this  session,  “Integration  of  the  Human 
Operator  into  Complex  Air  Systems  using  Alternative  Control 
Techniques,  by  Dr  Graham  Rood  of  DERA,  UK,  was  carried 
over  from  Session  III.  Demands  on  the  pilot,  in  addition  to  an 
ever-increasing  complexity  of  avionics  systems  and  the 
quantity  of  information  input,  include  the  frequent  need  for 
eyes-out  operation,  which  places  limits  on  the  practicability  of 
head-down  displays  and  switches.  Helmet-mounted  displays 
and  HOTAS  (hands  on  throttle  and  stick)  improve  pilot 
resptmse  in  such  situations,  but  the  large  number  of  functions 
and  control  modes  remains  a  problem.  Controls  and  switches 
per  crew  member  have  grown  in  number  from  single  digits  in 
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the  1920s  up  to  several  hundred  today,  a  trend  which  is  being 
repeated  in  HOTAS  switching  (up  to  40  functions  on  the  hand 
controls  of  some  aircraft),  with  associated  increase  in  the 
probability  of  error.  Further  problems  arise  from  variations  in 
physical  attributes  -  for  example,  gender  differences  in  the 
average  size  of  hands. 

Alternative  Control  Technologies  to  alleviate  these 
problems  were  discussed,  including  head  pointing,  eye 
tracking,  and  voice  control.  Direct  voice  input  (DVI),  though 
not  as  well  developed  as  the  foregoing  technologies,  can 
provide  a  quicker  and  more  natural  interface  and  has  proven 
effective  in  simulator  trials.  Longer  term  possibilities  are 
biopotential  and  gesture-based  control,  using  electromyo¬ 
graphic  (muscle  impulse)  or  electroencephalographic  (brain 
impulse)  inputs.  Finally,  the  development  of  unmanned  air 
vehicles  (UAVs)  was  mentioned,  which  would  profoundly 
change  the  nature  of  the  human  operator  interface.  To  a 
question  on  the  “best”  combination,  the  author  replied  that  it 
depended  ot  the  type  of  aircraft  -  fast  jets  for  example, 
required  speed  of  input/response  for  which  DVI  had  proven 
popular  in  tests  with  pilots  -  though  attention  had  to  be  paid 
to  noise  discrimination.  Other  questions  concerned:  the  areas 
of  the  brain  responsible  for  different  tasks  (on  which  work 
was  being  carried  out)  and  differences  in  flie  needs  and 
environment  of  UAV  operators  and  aircraft  pilots. 

SESSION  V:  LESSONS  LEARNED  AND  FUTURE 
TRENDS 

Chaired  by  Robert  Campbell,  US,  this  session  of  the  symp¬ 
osium  contained  two  papers  (two  papers  were  withdrawn). 

The  first  paper,  by  Bald  (Turidsh  Navy  Department  of 
Software  Development)  and  Kuru  (Bogazici  University),  TU, 
was  on  “Utihzing  CORBA  Concepts  for  Command  and 
Control  Systems”  and  presented  by  Metin  Bald.  The  paper 
described  an  evaluadon  of  the  CORBA  (Common  Object 
Request  Broker  Architecture)  as  a  platform-and-operadng 
system-independent  architecture  for  large  systems  such  as 
naval  C?.  A  background  descripdon  was  given  of  the 
requirements  for  a  military  system  and  its  software 
architecture,  leading  to  an  overview  for  distributed  systems, 
pardcularly  CORBA  and  common  object  services  built  on  the 
object  management  architecture  introduced  by  the  Object 
Managemant  Group  (OMG),  and  defined  in  an  interface 
definition  language  (IDL)  which  is  mapped  onto  other 
programming  languages  such  as  C,  C-H-,  Java,  Ada  and 
COBOL. 

An  architectural  evaluation  was  made  in  a  comparadve 
review  of  CORBA  and  altemadve  distributed  systems 
(DCOM,  RMI,  DCE)  against  a  number  of  criteria.  Perform¬ 
ance  assessments  were  carried  out  in  two  test  environments, 
the  latest  of  which  consists  of  four  Ultra-60  machines  with 
155MB  ATM  cormecdons.  In  summarising  the  evaluadon,  it 
was  concluded  that  CORBA,  as  currendy  specified,  though 
superior  to  the  altemadves  considered,  was  inadequate  for 
reliable  large-scale  C?  systems;  leading  to  the  decision  to 
build  an  intermediate  layer  on  top  of  a  CORBA  infrastructure. 
Related  research  on  CORBA  by  other  otganisadons  was  also 
reviewed,  nodng  particularly  OMG’s  plans  for  fault-tolerant 
improvements.  The  proposed  architecture  for  C^  systems 
applies  the  lessons  learned  from  the  evaluation  and  will  be 
implemented  in  prototype  form  using  COTS  products  as  far  as 
possible,  including  Java-based  programs  (especially  for  the 
user-interface).  The  author,  in  answer  to  quesdons,  provided 
clarification  of  the  architecture  and  of  the  CORBA  working 
group  (representing  government  and  uruversity  interests) 


which  will  lead  to  its  commerciaUsation. 

The  final  paper,  “The  Effects  of  Image  Resoludon  on  the 
Utihty  of  Target  Recognidon  Aids,”  by  Jan  Sdff  of  the  US 
Naval  Air  Warfare  Center,  China  Lake,  was  concerned  with 
imagery-based  target  recognidon  aids  (IBTRAs)  provided  to  a 
pilot  from  off-board  sources.  A  console,  used  by  the  thirty 
participants  in  the  experiments,  was  based  on  a  166MHz 
Pendum  computer,  with  digidsed  imagery  displayed  on  a 
17  inch  monitor  at  a  resolution  close  to  that  of  a  tacdcal 
aircraft  display.  Two  kinds  of  imagery  were  presented:  FUR 
imagery  from  exercise  and  trairring  flights;  and  reference 
imagery  which  was  derived  either  from  a  commercial,  SPOT 
source  (low  resoludon  -  10m)  or  from  US  Geologic  Survey 
aerial  recormaissance  informadon  (high  resoludon  -  Im).  Two 
images  were  presented,  the  FLIR  image  (close,  medium  or 
far)  and  the  reference  image  with  a  box  round  an  object  which 
the  test  subject  was  required  to  idendfy  in  the  FLIR  image. 

Results  were  given  in  terms  of  response  dme  (no  dme  limit 
imposed)  and  accuracy  (percentage  of  correct  responses)  for 
trials  at  the  three  ranges  and  varying  IBTRA  cross-range 
coverage.  Earlier  work,  showing  the  beneficial  effects  of 
matching  cross-range  extent  of  the  IBTRA  to  the  sensor 
image,  was  confirmed,  whilst  the  effect  of  resoludon  —  the 
main  purpose  of  the  present  study  -  appeared  to  be  more 
ambiguous.  It  was  concluded  that  beneficial  effects  of  high 
resoludon-derived  IBTRAs  were  most  likely  for  missions 
involving  weapon  release  close  to  the  target. 

SESSION  VI:  ROUND  TABLE 

The  round  table  discussion  which  concluded  the  sym¬ 
posium,  was  conducted  by  the  Chairman,  Ing  Pieter  van  den 
Broek,  as  an  open  forum  for  all  those  present.  He  got  the 
discussion  under  way  by  posing  the  quesdon:  how  important 
is  integradon  of  the  human  element?  There  exists  a  wide 
divergence  of  views  and,  as  he  pointed  out,  it  was  important 
to  recognise  that  the  human  element  has  its  own  failure 
modes.  Several  authors  had  commented  that  crews  involved  in 
trials  of  sensor  fusion  systems  felt  very  confident  with  them 
but  could  be  frustrated  when  a  fault  did  occur.  The  problem 
was  usually  handled  by  providing  extra  degrees  of  freedom, 
but  that  gave  rise  to  the  possibility  of  even  more  faults.  In 
general,  he  asked,  might  it  not  be  better  to  have  inadequate 
informadon  rather  than  false  informadon?  Other  contributors 
agreed  that  it  should  be  inherentiy  possible  for  systems  to 
provide  no  ident  or  low  probability  indications.  In  the  end 
however,  decisions  must  be  taken  by  the  system  with  - 
ultimately  -  human  operator  decision  on  appropriate  acdon. 
There  seem  to  be  no  ideal  soludcms  as  yet  but  the 
requirements  of  users  must  define  the  system  (including 
considerations  of  fratricide  and  pohdcal  issues).  At  present  it 
seems  best  to  continue  treating  fusion  systems  essentially  as 
operator  aiding  devices,  though  recognising  that  in  situations 
where  dme  is  vital,  the  system  might  have  to  be  taken  at  face 
value. 

Joseph  Ziegler  noted  that  lack  of  total  operator  confidence 
could  lead  to  a  desire  to  know  which  sensorfs)  are  contribut¬ 
ing,  though  it  would  be  difficult  to  present  all  the  informadon 
simultaneously  in  view  of  the  large  number  of  symbols 
required.  It  may  therefore  be  necessary  to  persuade  and/or 
train  operators  to  accept  such  systems  on  trust.  From  a  Navy 
point  of  view,  Medn  Bald  was  concerned  that  even  a  single 
failure  would  make  it  difficult  to  trust  the  system,  but  it  was 
pointed  out  that,  in  a  situation  when,  say,  three  sensors 
produced  agreed  results  supported  by  an  expert  system,  that 
ought  to  be  suffident.  Also,  it  was  reiterated  that  system 
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failures  could  be  detected  in  most  cases.  Graham  Rood 
observed  that  operators’  reaction  to  aiding  systems  was 
generally,  “when  we  can  see  it  woiics  better  than  we  do,  we 
will  accept  it!”  For  example,  terrain  avoidance  radar,  when 
first  introduced,  was  accepted  only  gradually;  it  was,  he 
agreed,  a  training  issue. 

Luigi  Crovella  raised  a  second  significant  question;  how  far 
can  fusion  systems  adapt  to  the  crew?  Graham  Rood  pointed 
out  that  voice  input  systems  are  already  a  two-way  process 
and  Ziegler  mentioned  that  systems  such  as  MIDS  do  have 
initialisation  inputs  from  the  human  operator.  Human  input  to 
modify  system  behaviour  is  also  a  desirable  feature  of  NATO 
AWACS  where  operators  want  to  apply  relative  weightings  to 
identification  inputs  -  evidence,  Crovella  remariced,  that  when 
engineers  talk  of  certainty,  operators  feel  uneasy.  Balci 
pointed  out  that  there  were  differences  between  ground  and 
ship  uses  for  AWACS  infonnation  and  on-board  confidence 
was  necessary  in  both  cases,  by  AWACS  as  well  as  by  the 
end-user. 

The  Chairman  returned  to  the  issue  of  operators’  confid¬ 
ence  and  their  natural  scepticism  regarding  the  rehability  of 
new  systems  and  wondered  why  it  was  necessary  to  have  an 
expensive  pilot  at  all  if  an  aircraft  were  fitted  with  an  all-glass 
cockpit.  He  rounded  off  the  discussion  by  commenting  that 
operators  may  use  new  systems  in  ways  that  were  not 
envisaged  by  those  (as  represented  by  the  round  table 
participants)  who  developed  them. 

CONCLUDING  REMARKS 

A  general  impression  was  that  the  papers  presented  were 
not  only  relevant  to  the  symposium’s  stated  theme  and  topics 
but  also  extremely  wide-ranging.  They  embraced  discussions 
of  the  basic  principles  of  sensor  data  fusion  and  operator 
behaviour  at  one  extreme,  to  presentations  of  results  from 
trials  of  complete  systems  at  the  other.  This  was  pe±aps 
reflected  in  answers  to  the  questioimaire  circulated  to 
participants;  more  than  half  of  the  respondents  replying 
positively  to  the  question;  has  the  symposium  provided 
valuable  new  perspective  in  your  work?  An  even  greater 
number  agreed  that  the  symposium  had  a  substantial  impact 
on  expanding  personal  contacts  for  potential  future 
collaboration;  indeed  this  was  listed  as  the  highest  priority 
aspect  of  the  symposium  by  most  participants.  This  view  can 
be  expected,  since  professional  peer  inter-communication  is 
generally  regarded  as  a  primary  reason  for  holding  symposia. 
More  to  the  point,  the  overall  value  of  the  symposium  was 
assessed  by  the  majority  as  at  least  “important  -  return  equals 
contribution"  v/ith  an  average  score  of  72  (on  a  scale  of  0  to 
100). 

The  essential  technical  concerns  of  the  symposium  -  the 
fusion  and  presentation  of  information  in  a  form  that  can  be 
easily  assimilated,  and  the  interaction  of  fusion  systems  with 
the  human  operator  -  was  well  covered,  as  was  the  fact  that 
advanced  sensors  and  communications  can  produce  a  deluge 
of  information  as  well  as  opportunities  for  improved  perform¬ 
ance.  The  problems  of  assimilating  large  amounts  of 
information  and  using  it  to  make  correct  decisions,  were 
shown  to  be  particularly  acute  in  the  case  of  command  and 
control  systems,  as  was  evident  in  several  papers.  These 
ranged  from  analysis  of  basic  C?  stmcmres,  to  computer  and 
software  requirements. 

The  opportunities  that  sensor  fusion  provides  for  new  or 
enhanced  missicai  capabilities  were  also  shown  in  a  variety  of 
papers,  covering  applications  as  diverse  as  multiple  target 
tracking,  and  helicopter  search  and  rescue  in  adverse  weather. 


Sensor  data  fusion,  to  enable  the  operator  to  make  sense  of 
multiple  sources  (or  combined  with  optical  sensors  in 
enhanced  vision  systems),  was  an  essential  element  in  all  of 
them.  It  was  also  salutary  to  hear  of  an  approach  in  the  field 
of  general  aviation  which,  though  constrained  by  consider¬ 
ations  of  size  and  cost  of  equipment,  was  attempting  to 
rationalise  cockpit  layout  so  as  to  reduce  pilot  workload.  For 
more  complex  applications,  several  papers  described  the 
development  of  operator  aids  based  on  cognitive  analysis  and 
applications  of  machine  intelligence  in  automated  or  semi- 
automated  systems. 

The  round  table  discussion  sounded  a  note  of  caution  on 
the  need  to  persuade  and  train  operators,  who  were  recognised 
to  be  naturally  cautious  of  new  technology  until  proven. 
Participants  also  recognised  the  importance  of  two-way 
interaction  between  system  and  operator,  to  provide  flexibihty 
of  response. 

In  conclusion,  the  symposium  provided  a  wide-ranging  and 
comprehensive  review  of  this  important  area  of  technology 
and  demonstrated  its  potential  for  major  improvements  in 
operators’  situation  awareness,  mission  effectiveness,  and 
survivability. 


Defense  R&D  in  Canada 


Following  are  the  viewgraphs  of  the  opening  address  for  the  symposium 
presented  by  Dr.  John  Leggat.  This  presentation  not  only  highlighted  the 
research  and  development  activities  in  the  host  country,  but  emphasised 
also  some  of  these  activities  related  to  the  subject  of  the  symposium. 
Because  of  this  the  presentation  is  included  in  the  proceedings  of  this 
symposium  in  the  form  of  these  (self-explaining)  viewgraphs. 


Pieter  Ph.  Van  den  Broek 
Acting  symposium  chairman. 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-I2. 
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■  The  Defence  R&D  Branch 

•  The  R&D  Program 

•  The  Strategy 

■  The  R&D  Network 

■  Some  Studies  Related  to  this  Symposium 

•  Environment  Perception  Process  in  Maritime  C&C 

•  Tactical  Information  Fusion  Prototype 

•  Decision  Support  System  in  Modem  Frigate 
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The  Defence  R&D  Branch 


Centre  of  Canadian  Defence  Technological  - 
Expertise 

Mission: 

•  Provide  expert  S&T  advice  for  decision  making 

•  Contribute  to  the  success  of  military  operations 

•  Enhance  the  preparedness  of  Canadian  Forces 

•  Facilitate  a  Canadian  defence  S&T  industrial 
capability 


Laboratories/Establishments 


Budget  $160M 
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Current  Core  Technologies 

■  Surveillance  and  Target  Acquisition 

■  Electronic  Warfare 

m  Command  Information  Systems 

■  Air  Vehicles 
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■  Naval  Platforms 

■  Combat  Systems 

■  Human  Systems  Integration  ^ 

■  Life  Support  Systems 
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Areas  of  Increasing  Emphasis 


■  Distributed  Interactive  Simulation 

■  Information  Warfare 

■  Signature  Reduction 

■  Non-Lethal  Weapons 

■  Advanced  Materials 

■  Biotechnology 

■  Robotics 
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Technology  Investment  Strategy 


TIS  Projects  forward  to  2010 

an  attempt  to: 

•  define  technology  competencies  we  must 
maintain,  withdraw  from  or  develop; 

•  identify  human  resources  and  skill  sets  to 
support  these  competencies; 

•  identify  a  framework  to  oversee  and 
manage  competency  developments  to 
realise  optimal  benefits  for  clients 


THE  R&D  NETWORK 


Defence  R&D  Branch  as  a  creator,  integrator 
and  enabler  in  the  areas  of  new  technology 


Partner  Programs 


International  Agreements 

•  The  Technical  Cooperation  Program  (TTCP) 

•  NATO  Research  and  Technology  Board 

•  Bilateral  Memoranda  of  Agreement 

Other  Canadian  Public  Sector  Institutes 

•  National  Research  Council 

•  Canadian  Space  Agency 

•  Provincial  Research  Agencies 

Industries  /  Universities 
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Partner  Programs 


■  Benefits  of  Partnership 

•  Leveraged  Resources 

•  Maintenance  of  critical  mass 

•  Enhanced  quality 

•  Quicker  progress 

•  Stable  working  relationships 

•  Commercial  exploitation 


SOME  STUDIES  RELATED 
TO  THIS  SYMPOSIUM 


Environment  Perception  Process  in  ! 
Maritime  Command  and  Controi 


■  Definition  of  an  architecture  for  data  fusion 
with  emphasis  to  situation  awareness  in  the 
context  of  dynamic  human  decision  making 

■  This  architecture  facilitates  the  proper  | 

conceptualisation  and  design  of  decision 
support  system  according  to  the  human  role  ! 
in  the  command  and  control  cycle 

■  Environment  Perception  Process  is  the 
foundation  of  the  architecture  for  data  fusion 


Various  operational  trends  in  naval  warfare,  such  as  technological  advances  in  threat  technology  and 
an  ongoing  shift  to  littoral  warfare,  put  the  shipboard  decision  making  process  under  pressure.  Data 
must  be  processed  under  time-critical  conditions  and  as  a  consequence,  the  risk  of  saturation  in 
building  the  tactical  picture  and  of  making  the  wrong  decision  increases. 

The  aim  of  this  study  is  to  explore  an  aspect  of  the  problem  of  dynamic  decision  making  in  the 
context  of  naval  command  and  control.  One  must  realise  that  the  human  plays  an  essential  role  in  the 
command  and  control  cycle.  Situation  awareness  is  essential  for  commanders  and  their  staff  to 
conduct  decision-making  activities.  Data  Fusion  is  seen  as  an  essential  process  to  enable  operators 
to  achieve  situation  awareness. 

The  study  is  a  step  towards  the  definition  of  an  architecture  for  data  fusion  giving  emphasis  to 
situation  awareness  in  the  context  of  dynamic  human  decision  making.  This  architecture  facilitates 
the  proper  conceptualisation  and  design  of  decision  support  system  according  to  the  human  role  in 
the  command  and  control  cycle. 


A  presentation  will  be  made  by  Mr  Stephane  Paradis  (DREV).  He  will  discuss  environment 
perception  process  as  the  foundation  of  this  architecture  for  data  fusion.  The  environment 
perception  process  is  aimed  at  achieving  a  first  level  of  situational  awareness,  which  forms  the  basis 
for  the  other  more  abstract  levels  of  situation  awareness.  Thus  the  quality  of  the  results  of  this  level 
is  of  utmost  importance  for  the  situational  awareness  that  can  be  gained  at  the  higher  levels. 


Tactical  Information  Fusion  Prototype 
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Decision  Support  System 
in  Modern  Frigate  [ 

■  Exploration  of  Concepts  for 

design,  development,  investigation,  and 
evaluation  of  computer-based,  real-time 
decision  support  system  (DSS)  to  be  | 

integrated  into  the  ship's  C2  Systems  to  assist 
operator  in  conducting  tactical  C2  | 

Operator  Serves  3  roles  in  the  C2  Process: 

•  situation  interpreter 

•  decision  maker 

•  effector 


DREV  is  exploring  concepts  for  the  design,  development,  implementation,  and  evaluation  of 
a  computer-based,  real-time  decision  support  system  (DSS)  that  can  be  integrated  into  the 
ship’s  CCS  to  assist  operators  in  conducting  tactical  C2.  The  operator  serves  three  primary 
roles  in  the  C2  process:  situation  interpreter,  decision  maker  and  effector.  The  purpose  of  the 
DSS  is  to  support  operators  in  each  of  these  roles  for  data  fusion  and  resource  management 
related  activities  in  Above-Water  Warfare  (AWW).  A  key  goal  is  the  design  of  a  joint 
system,  comprised  of  both  operators  and  automated  deeision  aids,  that  optimises  overall 
mission  performance,  leading  to  improved  operational  effectiveness. 

Developing  this  type  of  decision  support  system  is  a  challenging  task.  A  key  problem  is  that 
it  must  be  capable  of  operating  in  a  highly  dynamic  and  open  environment  that  imposes 
variable  and  unpredictable  demands  on  operators.  Operators  must  be  able  to  effectively 
handle  the  demands  of  new  and  unanticipated  situations  that  have  not  been  addressed  by  the 
system  designer  or  by  doctrine.  The  system  must  certainly  support  operators  so  that  they  can 
follow  established  principles  and  recommended  procedures.  Yet  it  must  not  overconstrain 
them  so  that  they  are  hampered  from  taking  advantage  of  their  abilities  to  reason,  improvise, 
and  respond,  while  at  the  same  time  calling  on  the  system  for  the  support  they  need. 


Synthetic  Displays  for 
Search  &  Rescue  Helicopters 


■ 


Investigate  the  use  of  an  HMD  system 
to  extend  the  operational  envelope  of 
search  and  rescue  helicopters  in 
day/night,  all  weather  situations 

System  will  include: 

synthetically  generated  image  based  on 
digital  map  databases  and  GPS  position 
information,  optically  fused  with  other 
sensor  images  (IR,  Visible) 
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Research  into  the  use  of  helmet  mounted  displays  (HMD)  is  being  carried  out  at  the  Flight 
Research  Laboratory  (FRL)  of  the  National  Research  Council  (NRC).  They  work  on  a 
collaborative  project  with  the  Canadian  Armed  Forces,  CAE  Electronics  and  the  Canadian 
Marconi  Company  to  investigate  the  use  of  an  HMD  system  to  extend  the  operational 
envelope  of  search  and  rescue  (SAR)  helicopters  in  day/night,  all  weather  situations  (100  ft 
ceiling,  l/8th  nm  visibility).  The  final  enhanced  and  synthetic  vision  system  (E/SVS), 
planned  for  testing  in  the  year  2000,  will  include  a  synthetically  generated  image  based  on  a 
digital  map  database  and  GPS  position  information,  optically  fused  with  other  sensor  images. 
The  sensor  images  could  include  an  IR  image  or  enhanced,  low-light  level  video.  Range 
information  will  be  provided  by  a  laser  range-finding  system.  The  system  must  provide  good 
situational  awareness  to  the  pilot  to  allow  operation  in  low  visibility  conditions  for  search  and 
rescue  operations  to  be  carried  out. 


Research  is  being  carried  out  at  FRL  into  the  system  requirements  of  HMD  systems  in 
helicopter  flight.  A  presentation  by  Mr.  Carl  Swail  will  include  results  to  date  and  a 
description  of  the  final  system.  He  will  discuss  human  factors  issues  related  to  the  integration 
of  the  different  sources  of  information.  The  present  research  system  consists  of  two  colour 
video  cameras  on  a  head-slaved  platform  mounted  on  the  roof  of  NRC's  Bell  205  helicopter 
and  two  video  projection  systems  inside  the  helicopter  projecting  images  through  fibre  optic 
bundles  to  the  helmet  mount  display  optics.  Baseline  handling-qualities  investigations  have 
been  carried  out  to  determine  the  effects  of  using  an  HMD  on  the  pilot's  workload  and 
performance  while  performing  standard  manoeuvres. 


mportant  Points 


Multi-sensor  landmine  detection  system  was 
shown  to  reduce  false  alarm  rate  while  retaining 
acceptable  Probability  of  Detection 
Temporal  and  spatial  correlation  of  data  is  crucial 

Sensors  must  be  ^'orthogonal"  -  i.e.  measure 
independent  properties  for  data  fusion  to  work 

Datasets  are  very  expensive  to  generate 
Difficult  to  develop  surrogate  targets  (ie  which 
test  all  sensors  at  same  time) 
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SENSOR  DATAFUSION  TECHNOLOGIES  FOR  PRECISION  STRIKE 
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Seal  Beach,  CA  90740  USA 
Telephone:  (562)  797-2382 
Fax:  (562)  797-4854 
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SUMMARY 

This  paper  provides  an  assessment  of  the  enabling 
technologies  for  sensor  data  tusion.  The  assessment  is 
conducted  in  the  context  of  precision  strike  for  the  year 
2010  time  frame.  Emphasis  is  given  to  the  sensor  data 
ftision  technologies  that  reduce  the  workload  and  improve 
the  performance  and  safety  of  the  human  element. 
Technology  advances  are  projected  for  C4ISR,  sensors, 
electronics,  and  weapons  that  enable  sensor  data  fusion, 
with  the  operational  considerations  of  weather  and  clutter. 

Preferred  sensors  are  synthetic  aperture  radar  (SAR), 
passive  imaging  millimeter  wave  (mmW),  active  imaging 
infrared  (HR),  and  passive  HR. 

Examples  are  shown  of  robust  sensor  data  fusion 
architecture  for  high  detection  and  low  false  alarm. 

Follow-on  system-level  studies  to  assess  sensor  data 
fusion  concepts  and  technologies  are  outlined. 

INTRODUCTION 

Sensor  data  fusion  has  numerous  military  applications, 
including  the  precision  strike  mission  area.  Precision 
strike  requires  human  elements  such  as  image  analysts 
and  mission  planners  involved  in  command,  control, 
communication,  computers,  information,  surveillance, 
reconnaissance  (C4ISR)  and  pilots  involved  in  weapon 
delivery.  Benefits  of  sensor  data  fusion  for  the  human 
element  include  reduced  workload,  improved  throughput, 
and  improved  accuracy.  Sensor  data  fusion  also  improves 
safety  of  the  pilot  and  friendly  forces/civilians  in  the  target 


area.  Figure  1  shows  the  benefits  of  sensor  data  fusion  for 
the  precision  strike  mission  area. 

Although  an  autonomous  system  does  not  have  the 
human  frailties  of  fatigue  or  variation  in  performance  over 
time,  autonomous  systems  are  not  foolproof  In  the 
foreseeable  future,  sensor  data  fusion  algorithms  are 
unlikely  to  embody  the  capability  of  the  human  element 
to  resolve  ambiguities  and  errors  in  target  Identification. 
Image  analysts,  mission  planners,  and  pilots  will  provide 
oversight  monitoring  to  ensure  that  correct  targets  are 
selected.  Sensor  data  fusion  will  provide  a  leveraging  of 
human  efficiency,  rather  than  a  replacement  of  the  human 
element. 

Figure  2  illustrates  the  need  for  an  automated  system  to 
conduct  the  image  analysis  used  in  C4ISR.  For  a  non- 
automated  system,  the  number  of  image  analysts  required 
to  conduct  daily  area  coverage  of  a  threat  country  the 
size  of  Iran  would  be  unrealistically  large.  As  an  extreme 
example,  more  than  1 00,000  image  analysts,  examining 
a  full  frame  of  1/3  meter  resolution  data  every  two 
minutes,  would  be  required  to  evaluate  daily  coverage  of 
Iran.  An  impossible  task!  In  this  case  the  image  analyst  is 
drovming  in  data  but  starved  for  information.  The  desired 
analysis  for  a  data  rate  of  3  Gbps  can  be  accomplished 
only  by  automating  the  system.  Figure  2  also  shows  the 
search  and  the  spotlight  data  rates  of  the  Global  Hawk 
unmanned  air  vehicle  (UAV).  Note  that  improvements 
are  also  required  in  data  rate,  due  to  the  limited  size  of 
the  UAV  force. 
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Figure  1.  Benefits  of  Sensor  Data  Fusion  for  Precision  Strike 
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Figure  2.  Automated  Systems  are  Required  for 
C4ISR  Image  Analysis 


An  example  of  the  importance  of  timely,  high-resolution 
data  is  the  problem  of  countering  threat  theater  ballistic 
missiles  (TBMs).  TBMs  have  characteristics  of  short 
timelines,  low  signature,  and  high  mobility  associated 
with  a  “hide-scoot-shoot-hide”  doctrine.  Events  such  as 
leaving  a  camouflage/concealment  site,  moving  to  a 
launch  site,  preparing  for  launch,  tearing  down  after 
launch,  and  moving  to  a  camouflage/concealment  site  are 
often  not  observed  or  they  are  observed  too  late  to  provide 
an  effective  counterforce  response.  New  sensor  data 
fusion  technologies  are  required  to  expedite  the  C4ISR 
process  for  time  critical  targets  such  as  TBMs. 

Sensor  data  fusion  also  enhances  pilot  safety.  Some 
missions,  such  as  low  altitude  weapon  delivery,  may  be 
too  dangerous  to  put  pilots  at  risk  and  may  be  best  handled 
by  autonomous  weapons  such  as  cruise  missiles.  The 
problem  is  illustrated  in  Figure  3,  which  shows  the  anti¬ 
aircraft  artillery  (AAA)  fire  over  Baghdad  during  the  first 
night  of  Operation  Desert  Storm. 


Figure  3.  Automated  Systems  Avoid  Putting  Pilots 
in  Harm’s  Way 


Although  the  U.S.  Department  of  Defense  (DoD)  does 
not  have  a  technology  area  dedicated  to  sensor  data  fusion, 
the  technology  areas  associated  with  sensor  data  fusion 
receive  high  emphasis  in  the  DoD  technology  funding. 
The  DoD  top-three  military  technology  areas  are  the 
following:  information  systems,  weapons,  and  sensors/ 
electronics  (Figure  4).  The  funding  for  these  areas  is 
greater  than  the  combined  total  of  the  other  military 
related  technologies.  Sensor  data  fusion  is  emphasized 
in  the  information  systems,  weapons,  and  sensors/ 
electronics  technology  areas. 

An  example  of  the  increased  application  of  sensor  data 
fusion  is  combat  aircraft  avionics.  Combat  aircraft 
avionics  have  been  increasing  in  emphasis  each  year. 
Avionics  has  now  become  the  dominant  consideration  in 
combat  aircraft.  In  the  1960s,  aircraft  such  as  the  F-4 
had  relatively  low  consideration  of  avionics,  with  only 
about  10%  of  the  flyaway  cost  allocated  to  avionics 
(Figure  5).  The  dominant  costs  for  the  F-4  aircraft  were 
engines  and  airframe.  Although  the  aeromechanics 
performance  of  combat  aircraft  has  had  only  evolutionary 
advancements,  the  advancements  in  avionics  have  been 
revolutionary.  Each  new  aircraft  has  brought  an  increasing 
emphasis  in  avionics,  due  primarily  to  improvements  in 
the  DoD  Top  3  technology  areas  of  information  systems, 
weapons,  and  sensors/electronics.  An  emerging  market 
for  older  aircraft  is  the  periodic  updating  of  their  avionics. 
As  shown  in  Figure  5,  the  cost  of  future  aircraft  will  be 
primarily  driven  by  their  avionics. 

An  application  of  sensor  data  fusion  is  the  use  of  ATR  to 
accurately  detect  and  identify  targets  in  a  computationally 
efficient  and  timely  manner.  Challenges  include  weather, 
clutter,  and  changes  in  the  target  signature.  Changes  in 
the  target  signature  could  include  target  wear,  orientation, 
and  camouflage.  Although  no  ATR  system  is  perfect, 
sensor  data  fusion  provides  robustness  in  false  alarm  rate 
and  detection. 

Examples  of  a  broad  range  of  alternative  systems  that  use 
sensor  data  fusion  are  shown  in  Figure  6.  Going  clockwise 
around  the  figure,  the  sensor  systems  tend  to  have 
decreasing  range,  sophistication,  and  cost: 

•  Predator  unmanned  air  vehicle  (UAV)  surveillance 
system.  Predator  has  synthetic  aperture  radar  (SAR), 
passive  imaging  infrared  (HR),  and  visible  electro- 
optical  (EO)  sensors. 

•  Low  Altitude  Navigation  and  Targeting  Infrared  for 
Night  (LANTIRN)  targeting  pod  for  combat  aircraft. 
LANTIRN  fuses  passive  HR  and  laser  detection  and 
ranging  (LADAR)  sensor  data. 
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Figure  4.  Sensor  Data  Fusion  is  Reflected  in  the  U.S.  DoD  Top  3  Technology  Areas 


Source; 

•  D.  Morgan,  “The  Future  of  Avionics  Architecture,”  Presented 
at  AGARD  Conference  on  Advanced  Architectures  for 
Aerospace  Mission  Systems,  October  1996 

•  J.  Lupo,  “DDR&E  Perspective  on  Avionics  Technology,"  April 
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Figure  5.  Avionics  Has  Become  Very  Important  to 
Fighter  Aircraft 

•  AGM-130  missile  guidance  and  control.  AGM-1 30  fuses 
global  positioning  satellite  (GPS)  receiver/inertial 
navigation  system  (INS)  and  passive  HR  seeker  sensor 
data  for  precision  guidance. 

•  Brilliant  Anti-Tank  (BAT)  guided  submunition.  BAT 
fuses  acoustic  and  passive  HR  sensor  data  for 
autonomous  acquisition  and  strike  of  mobile  targets. 

The  above  sensor  systems  use  a  wide  field  of  view  (FOV) 
sensor  for  target  detection  and  a  narrow  FOV  sensor  for 
target  identification  and  targeting.  The  wide  FOV  sensor 
cues  the  narrow  FOV  sensor. 

Sensor  data  fusion  was  defined  in  1987  by  the  Joint  DoD 
Laboratories  (JDL)  subpanel  on  Data  Fusion  as  “the 


process  dealing  with  the  association  of  sensor  data  and 
the  estimation  of  entity  kinematics,  attributes,  and 
identity  to  achieve  assessments  and  projections  of  a 
situation.”  Sensor  data  fusion  combines  different  sets 
of  data  from  one  or  more  sensors.  Fusion  can  occur  over 
multiple  wavelengths,  space,  or  time.  As  the  data  channels 
are  combined,  the  processing  evolves  from  signal 
processing  to  information  processing,  resulting  in 
information  of  higher  value  to  the  human  element. 

SENSOR  DESIGN  CONSIDERATIONS  FOR 
PRECISION  STRIKE 

The  precision  strike  design  considerations  of  weather, 
clutter,  C41SR,  target  set,  sensor/processor  performance 
and  cost,  and  navigation  accuracy  will  be  discussed  in 
this  section. 

Weather,  Including  cloud  cover  and  rain,  is  a  driving 
consideration  in  the  selection  of  sensors.  As  an  example 
of  the  effect  of  weather  on  IR  sensors,  IR  sensors  are 
blocked  if  a  cloud  masks  the  target  and  IR  sensor 
performance  is  degraded  by  rain. 

Clouds  are  pervasive  in  the  world  wide  weather.  The 
average  global  annual  cloud  cover  is  about  61  percent 
(Figure  7),  with  an  average  cloud  cover  over  land  of  about 
52  percent  and  an  average  cloud  cover  over  the  oceans 
of  about  65  percent.  Clouds  tend  to  occur  at  a  height 
between  3,000  and  20,000  feet.  Due  to  the  frequency  of 
low  cloud  cover,  airborne  IR  sensors  are  often  limited  in 
their  applications  to  flight  at  heights  of  less  than  3,000 
feet. 

Figure  8  shows  the  signal  attenuation  versus  wavelength 
for  a  representative  cloud,  rain  rate,  and  humidity  level 
requirement.  Note  that  although  passive  EO  sensors  have 
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Figure  6.  Example  of  Alternative  Sensors  for  Data  Fusion 


Figure  7.  Cloud  Cover  is  Pervasive 


one-way  transmission  through  rain  of  about  50  percent 
per  km,  there  is  almost  no  transmission  of  an  EO  signal 
through  clouds.  Cloud  droplets  are  small  -  about  5  to  20 
microns  in  diameter,  with  dimensions  comparable  to  the 
EO  wavelength.  The  concentration  is  high — about  50  to 
500  droplets  per  cubic  centimeter.  EO  wavelengths  are 


strongly  diffracted  around  cloud  droplets  due  to  Mie 
electromagnetic  scattering.  However,  raindrops  are  about 
2-6  mm  in  diameter  (much  larger  than  EO  wavelengths) 
and  cause  less  attenuation  to  an  EO  signal.  Rain  rate 
attenuation  is  due  primarily  to  optical  scattering.  EO 
transmission  through  rain  is  a  function  of  the  size  of  the 


Attenuation  by  Atmospheric  Gases,  Clouds,  And  Rain  (From  Priessner,  1979) 
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Figure  8.  Signal  Attenuation  Due  to  Weather 


raindrops,  rain  rate,  and  the  path  length  through  the  rain.  example,  a  low  frequency  ultra  wide  band  SAR  sensor  has 
EO  passive  sensors  are  limited  from  about  2  to  5  km  of  foliage  penetration  that  is  not  possible  with  higher 
path  length  through  rain.  frequency,  shorter  wavelength  sensors. 


The  best  sensors  in  looking  through  clouds  and  rain  are 
radar  sensors.  Radar  sensors  have  negligible  attenuation 
at  frequencies  below  10  GHz.  At  higher  frequencies, 
millimeter  wave  sensors  operating  in  clouds  and  rain  are 
limited  from  about  2  to  5  km  length  of  path  through  the 
clouds  and  rain,  with  the  same  implications  as  those 
discussed  in  the  previous  paragraph.  Cloud  droplets, 
which  are  much  smaller  than  mmW  wavelengths,  absorb 
mmW  radiation  (much  like  a  microwave  oven).  A  different 
mechanism  is  responsible  for  the  attenuation  of  a  mmW 
signal  through  rain  or  snow.  Raindrops  and  snowflakes 
are  comparable  in  size  to  mmW  wavelengths  and  cause 
Rayleigh  and  Mie  electromagnetic  scattering  attenuation. 
Lower  frequency,  longer  wavelength  Ku-band  sensors  are 
less  affected  by  cloud  cover  and  rain/snow. 

The  second  consideration  to  be  discussed  for  the 
precision  strike  mission  is  clutter.  Like  weather,  clutter 
is  pervasive —  everything  that  is  not  part  of  the  target  is 
clutter.  There  are  two  types  of  clutter,  natural  clutter  and 
man-made  cultural  clutter.  Natural  clutter  includes 
terrain,  lakes,  rivers,  trees,  bushes,  plants,  and  grass. 
Cultural  clutter  includes  buildings,  roads,  and  vehicles. 
Clutter  complicates  ATR  because  clutter  can  look  like  a 
target,  and  targets  can  be  partially  or  completely  obscured 
by  clutter.  A  benefit  of  sensor  data  fusion  is  that  clutter 
for  one  type  of  sensor  or  discriminant  may  be  less 
limiting  to  another  type  of  sensor  or  discriminant.  For 


In  high  clutter  areas  (e.g.,  urban,  villages  crowded 
together,  forests,  mountains,  jungles,  brown  water),  the 
threat  may  rely  on  the  exploitation  of  confusing  clutter. 
In  an  urban  area,  targets  may  be  at  street  level,  on  roofs 
and  the  upper  stories  of  buildings,  and  below  street  level 
in  bunkers,  sewer  systems,  subways,  and  other 
underground  structures.  A  particular  concern  is  precision 
strike  for  military  operation  in  urban  terrain  (MOUT). 
MOUT  has  a  risk  for  confusion  of  civilian  and  friendly 
force  assets  with  the  threat  military  assets.  Figure  9 
illustrates  a  MOUT  environment. 


A  third  design  consideration  for  precision  strike  is  the 
C4ISR  architecture.  Figure  10  illustrates  an  example  of 
C4ISR  architecture  of  the  year  2010  time  frame.  It 


Figures.  Example  of  MOUT  Clutter 


K-6 


Figure  10.  Example  of  C4ISR  Targeting  • 


includes  a  ground  station,  overhead  low  cost  tactical 
satellite  sensors  (e.g.,  Starlite/Discoverer  II),  satellite 
relays,  and  overhead  UAV  sensor  platform  elements.  The 
C4ISR  architecture  of  the  year  2010  is  projected  to  have 
a  capability  of  detecting  targets  in  clutter,  target  location 
error  (TLE)  of  less  than  1  meter  (1  sigma)  and  off-board 
sensor-to-shooter  connectivity  time  of  less  than  2 
minutes  (1  sigma).  This  capability  will  enable  a 
counterforce  response  against  time  critical  targets,  such 
as  TBMs. 

Target  sensors  of  the  year  2010  time  frame  will  use  real¬ 
time  ATR  for  precision  targeting.  ATR  performance  will 
be  enhanced  through  multidimensional  orthogonal 
discriminants.  The  best  overall  target  sensors  for 
precision  strike  in  adverse  weather  and  clutter  are 
considered  to  be  X-band  or  Ku-band  SAR  sensors  and 
passive  imaging  mmW  sensors.  Other  sensors  for 
precision  strike,  listed  in  order  of  their  expected 
application  in  the  year  2010,  include  active  imaging  IR 
based  on  LADAR,  passive  imaging  IR,  active  nonimaging 
radar,  active  nonimaging  LADAR,  passive  nonimaging  IR, 
visible,  UV,  and  acoustic/seismic.  Sensors  are  available 
that  cover  the  range  of  acoustic-to-visible.  However,  due 
to  the  importance  of  standoff  in  adverse  weather,  radar 
and  mmW  will  be  the  primary  sensors  considered  for 
the  precision  strike  mission  area.  Multidimensional 
orthogonal  discriminants  that  are  likely  to  be  used  in 
sensor  data  fusion  include  angular  resolution,  range 
resolution,  contrast,  polarization,  temporal  (e.g., 
vibration,  movement),  multispectral,  and  gas  composition 
signatures  of  the  target. 

Until  recently,  most  UAVs  had  relatively  unsophisticated 
sensor  suites  that  typically  consisted  of  a  forward  looking 
infrared  (FLIR)  sensor,  a  television  (TV)  camera,  and  a 
data  link.  However,  the  more  recent  UAVs,  such  as 
Predator,  are  beginning  to  carry  sophisticated  sensor 
payloads,  including  multiple  IR  sensors  and  SAR  sensors. 


that  broaden  the  range  of  missions  and  mission 
performance.  Improvements  in  sensor  capability  are 
expected  to  continue  into  the  next-generation 
multimission  UAVs.  Drivers  for  the  next-generation 
UAVs,  which  will  result  in  improved  sensor  capability, 
include  a  broader  range  of  missions,  enhanced 
survivability,  wide  area/continuous  coverage  of  the  target 
area,  real  time  ATR,  low  False  Alarm  Rate  (FAR),  and 
real-time  precision  targeting. 

The  General  Atomics  Predator  UAV  is  an  example  of 
recently  introduced  UAVs.  Developmental  UAVs  such 
as  Global  Hawk  and  Dark  Star  will  be  more  sophisticated 
than  Predator.  The  Predator  sensor  payload  includes  an 
EO  sensor  suite,  a  Ku-bahd  SAR  sensor,  Ku-band  and 
UHF-band  satellite  communication  (SATCOM),  a  C-band 
line-of-sight  data  link,  and  a  GPS/  fNS  navigator. 

The  Predator’s  SAR  sensor  is  the  Northrop  Grumman 
(Westinghouse)  Tactical  Endurance  Synthetic  Aperture 
Radar  (TESAR).  TESAR  provides  continuous,  near  real 
time  strip-map  transmitted  imagery  over  an  800  meter 
swath  at  slant  ranges  up  to  1 1  km.  Maximum  data  rate  is 
500,000  pixels  per  second.  The  target  resolution  is  0.3 
meters.  TESAR  weight  and  power  are  80  kg  and  1200 
watts  respectively.  A  lighter  weight,  lower  cost  SAR  is 
currently  in  development  for  Predator. 

The  Predator’s  EO  sensor  suite  is  the  Versatron  Skyball 
SA-144/18  quartet  sensor.  It  consists  of  a  PtSi  512  x 
512  MWIR  FLIR  with  six  fields  of  view,  a  color  TV 
camera  with  a  lOX  zoom,  a  color  TV  900  mm  camera, 
and  an  eyesafe  pulsed  erbium:  glass  laser  range  finder. 
The  diameter  of  the  EO  sensor  turret  is  relatively  small  - 
35  cm.  The  turret  has  precision  pointing  with  a  line-of- 
sight  stabilization  accuracy  of  10  prad. 

It  is  anticipated  that  high  performance  UAVs  of  the  year 
2010  will  have  a  broad  range  of  missions,  including 
surveillance,  reconnaissance,  communication, 
intelligence  gathering  of  threat  electronic  emissions, 
target  designation  for  weapons  attacking  moving  targets, 
and  communication  relay.  The  system  flight  performance 
is  expected  to  be  greater  than  present  UAV s,  with  higher 
velocity  providing  a  larger  area  of  coverage  and  faster 
response  in  getting  to  the  target  area.  Higher  cruise 
altitude,  above  60,000  feet,  will  provide  a  longer  line- 
of-sight  data  link  (Figure  11)  and  safer  operation  above 
commercial  aircraft.  As  an  example  of  the  current  state- 
of-the-art  (SOTA)  in  data  rate.  Global  Hawk  has  a  data 
rate  of  approximately  1.6  million  pixels  per  second  at  a 
resolution  of  1  meter.  The  daily  data  rate  is  approximately 
30  Mbps  in  a  1 -meter-resolution  search  mode  and 
approximately  10  Mbps  in  a  1/3-meter-resolution 
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Figure  11.  High  Altitude  UAVs  Provide  Long  Range 
C4ISR  and  Safe  Operation  Above  Commercial  Aircraft 

spotlight  mode  (Figure  2).  However,  this  does  not  provide 
full  daily  coverage  of  a  large  country  such  as  Iran.  Year 
2010  UAVs  will  have  higher  data  rate  and  the  implication 
of  automated  image  analysis  and  sensor  data  fusion  for 
ATR.  Finally,  the  precision  pointing  and  line-of-sight 
stabilization  accuracy  of  future  UAVs  will  be  improved 
to  about  5  |Lirad,  to  accommodate  longer  standoff. 

Advancements  in  sensor  capability  for  C4ISR  will  provide 
new  capabilities  of  near  real  time  ATR,  orders  of  magni¬ 
tude  reduction  in  FAR,  an  order  of  magnitude  improve¬ 
ment  in  targeting  accuracy,  and  an  order  of  magnitude 
improvement  in  data  rate.  The  advanced  sensors  will 
leverage  the  current  advanced  development  (category 
6.3  A  funding)  activities  that  are  presently  under  way. 

There  is  emphasis  to  reduce  the  logistics  cost  by 
producing  multipurpose  systems  that  cover  a  broad  range 


of  targets.  An  example  is  the  Joint  Standoff  Weapon 
(JSOW).  JSOW  is  a  neck-down  replacement  of  Walleye, 
Skipper,  Rockeye,  laser  guided  bombs.  Maverick,  and  other 
precision  strike  weapons.  A  multipurpose  weapon  system 
for  precision  strike  must  consider  a  broad  target  set  (Figure 
12).  Challenges  include  targets  with  low  signature  (e.g., 
command  and  control  bunkers),  small  size  (e.g.,  trucks), 
small  vulnerable  area  (e.g.,  armor),  mobility  (e.g.,  artillery), 
time  urgency  (e.g.,  missile  transporter/erector/launcher), 
and  weapon  cost  effectiveness  (e.g.,  cost  per  kill  versus 
target  value)  considerations.  ATR  for  a  multipurpose 
weapon  system  against  the  broad  range  of  targets  is 
enhanced  through  sensor  data  fusion,  exploiting  the 
unique  characteristics  of  each  target. 

The  revolutionary  advancement  of  high  performance,  low 
cost  processors  is  an  enabling  technology  for  sensor  data 
fusion.  The  capability  to  process  mutidimensional 
discriminants  at  low  cost,  small  size,  and  low  power  is 
just  beginning  to  emerge.  Figure  13  shows  the  history  of 
processing  capability  of  commercial  chips,  with  a 
projection  to  the  year  2010.  Note  that  for  the  last  twenty 
three  years  the  processing  capability  has  been  doubling 
about  every  two  years,  going  from  2,300  transistors  on 
the  4004  chip  in  1972  to  5.5  million  transistors  on  the 
Pentium  Pro  chip  in  1995.  There  is  no  sign  that  the 
growth  rate  will  slow  down.  A  projection  to  the  year  20 1 0 
predicts  a  capability  of  over  1  billion  transistors  on  a 
chip.  Processing  capability  is  ceasing  to  be  a  limitation 
for  the  application  of  sensor  data  fusion  to  precision 
strike.  The  limitations  for  precision  strike  are  now 
focused  on  ATR  algorithms,  cost  effective  sensors,  and 
sensor  data  fusion  architecture. 


Figure  12.  Precision  Strike  Addresses  a  Broad  Range  of  Targets 
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Figure  13.  Revolutionary  Advances  in  Processing 
Enables  Sensor  Data  Fusion 


The  revolutionary  advancement  described  above  in  the 
performance  of  low  cost  commercial  chips  has  led  to  the 
recent  movement  to  incorporate  commercial  parts  into 
military  products.  Figure  14  shows  the  decreasing  presence 
of  military  parts  in  the  integrated  circuit  (IC)  market  and 
the  decline  of  military  procurement.  The  military  IC 
presence  has  diminished  to  where  it  is  now  less  than  1 
percent  of  the  IC  market.  At  this  level,  the  military  exerts 
almost  no  Influence  on  IC  manufacturers  to  supply  the 
technology  needed  for  military  electronic  parts.  The 
relatively  low  level  of  military  chip  procurement  has  caused 
several  traditional  high-reliability  IC  manufacturers  to 


Figure  14.  COTS  Electronics  Are  Low  Cost  Because 
Commercial  Sales  Are  Driving  Semiconductors 


leave  the  military  market.  The  result  is  less  selection  of 
military-type  parts.  As  this  trend  continues,  military 
electronics  will  use  low-cost  commercial  plastic  parts  as 
the  norm.  The  low-cost  commercial  plastic  parts  will  have 
to  be  cocooned  for  protection  against  the  harsher  military 
environment  (temperature,  vibration,  acoustics,  moisture, 
dust,  salt,  etc.).  New  technology  is  required  to  develop 
cocoons  for  the  commercial  off  the  shelf  (COTS)  parts. 
Another  consideration  is  the  rapid  introduction  of  new 
commercial  products.  Due  to  the  relatively  long 
development  time  of  military  products  compared  to 
commercial  products,  the  military  electronics  are  often  one- 
to-three  generations  behind  the  commercial  state-of-the- 
art  (SOTA)  by  the  time  the  military  product  is  introduced. 
In  addition,  the  military  product  may  not  have  a  logistical 
support  base  due  to  obsolete  parts.  This  requires  the 
consideration  of  a  robust  architecture  to  handle  frequent 
upgrades  of  the  commercial  parts.  The  frequent  upgrades 
will  require  a  streamlined  process  for  military  qualification 
tests,  to  make  the  more  frequent  qualification  tests 
affordable. 

Another  example  of  a  commercial-type  technology  that  is 
reducing  sensor  cost  is  Micro  Electro  Mechanical  Systems 
(MEMS).  With  MEMS  technology,  inertial  measurement 
units  (IMUs)  may  cost  $2,000  to  $3,000  in  the  year  2010 
time  frame.  MEMS  devices  are  fabricated  from  a  single 
piece  of  silicon  by  commercial-type  semiconductor 
manufacturing  processes,  resulting  in  a  small,  low-cost 
package  (Figure  15).  The  compatibility  with  high  rate 
production  and  broad  range  of  applications  (military  and 
commercial)  are  enablers  for  low  cost.  Between  2,000  and 
5,000  MEMS  gyro  devices  can  be  produced  on  a  single 
five-inch  silicon  wafer. 


Precision  GPS/INS  guidance  is  an  enabling  consideration 
for  sensor  data  fusion  in  high  clutter.  The  current  GPS 
receivers  operating  in  a  military  P(Y)  code  have  an  accuracy 


Figure  15.  MEMS  is  Small,  Lightweight,  and 
Low  Cost 
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of  approximately  6  meters.  Recent  advances  in  using  GPS 
in  the  Wide  Area  GPS  Enhancement  (WAGE)  mode, 
differential  mode,  or  relative  mode,  combined  with  SAR 
precision  mapping,  provide  errors  less  than 
3  meters. 

Figure  16  illustrates  the  benefit  of  precision  GPS/INS  guid¬ 
ance  for  a  bridge  target  in  a  high  clutter  environment  con¬ 
sisting  of  a  river  and  rolling  terrain.  In  this  example,  a  high- 
resolution  (640  pixels  x  480  pixel)  HR  sensor  is  used  for 
terminal  homing  for  precision  strike  of  the  center  pier  of 
the  bridge.  Note  that  even  for  a  relatively  large  target  such 
as  the  bridge  in  the  figure,  there  is  a  large  amount  of  clut¬ 
ter  at  a  seeker  lock-on  range  of  850  meters.  The  20-degree 
FOV  seeker  provides  a  300  meters  x  230  meters  scene  with 
high  resolution  (1  pixel  =  0.47  meters).  The  GPS/INS  error 
is  assumed  to  be  3  meters,  resulting  in  a  6  pixel  correction 
error  for  a  seeker  lock-on  range  of  850  meters.  The  maxi¬ 
mum  acceleration  to  correct  the  3-meter  initial  heading 
error  is  small,  about  1 .0  g.  Fortunately,  the  GPS/INS  preci¬ 


sion  accuracy  (3  meters)  allows  delay  of  seeker  lock-on 
and  terminal  guidance  until  the  missile  is  closer  to  the 
target.  As  the  missile  closes  on  the  target  at  the  seeker 
lock-on  alternative  ranges  of  500  meters  and  250  meters, 
note  that  there  is  less  and  less  clutter.  The 
maneuveracceleration  to  correct  the  3-meter  initial  head¬ 
ing  error  is  1.7  g’s  and  3.5  g’s  respectively.  Finally,  at  a 
range-to-go  of  125  meters,  note  that  there  is  almost  no 
clutter  within  the  seeker  FOV.  Relatively  high  maneuver¬ 
ability  is  required  (7.1  g’s)  to  correct  the  GPS/INS  initial 
heading  error  of  3  meters.  The  GPS/INS  accuracy  of  3 
meters  is  comparable  to  the  tracking  accuracy  of  the  seeker, 
resulting  in  a  relatively  smooth  transition  in  handing  off 
from  midcourse  navigation  GPS/INS  guidance  to  terminal 
seeker  guidance.  For  a  typical  closing  velocity  of 300  meters 
per  second  and  a  guidance  update  rate  of  60  Hz,  there  are 
25  guidance  updates  available  to  conduct  ATR  for  a  seeker 
lock-on  range-to-go  of  125  meters.  The  terminal  accuracy 
is  more  a  ftinction  of  the  airframe  response  (assumed  to  be 
0. 1  second)  than  that  of  seeker  tracking  error.  As  a  result 
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Figure  16.  Precision  GPS/iNS  Facilitates  Targeting  in  a  Clutter 
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of  the  GPS/INS  precision  guidance,  the  missile  has  higher 
probability  of  hitting  the  center  pier  aim  point  and  drop¬ 
ping  the  center  span  of  the  bridge.  The  GPS/INS  guided 
missile  can  be  treated  almost  like  an  artillery  projectile  of 
low  ballistic  dispersal,  with  high  probability  of  impacting 
in  the  target  area  and  low  probability  of  collateral  damage. 

Benefits  of  tightly  coupled  integration  of  INS  with  GPS 
include  high-precision  position  and  velocity 
measurement  and  reduced  jamming  susceptibility.  The 
availability  of  GPS  to  continuously  update  the  inertial 
system  allows  the  design  trades  to  consider  a  lower 
precision  and  less  expensive  INS,  while  maintaining  good 
navigation  accuracy  and  anti-jam  (A/J)  performance. 

Future  GPS/INS  receivers  will  be  based  on  a  centralized 
Kalman  filter  that  processes  the  raw  data  from  all  of  the 
sensors  (e.g.,  SAR,  GPS  receiver,  INS).  Tightly  coupled 
GPS/INS  is  more  robust  against  jamming  because  it  is  able 
to  make  pseudo-range  measurements  from  three,  two,  or 
even  one  satellite  if  one  or  more  of  the  satellites  are  lost. 

The  conservative  6-meter  GPS  standard  accuracy  can  be 
significantly  improved.  An  example  shown  in  Table  1  is  a 
Conventional  Air  Launched  Cruise  Missile  (CALCM) 
demonstration  based  on  Precision  On-Board  GPS 
Optimization  (POGO).  POGO  demonstrated  precision 
accuracy  using  GPS-aided  inertial  navigation.  POGO  uses 
smart  algorithms  to  reduce  satellite  clock  errors  and  satellite 
ephermis  errors.  The  use  of  an  all-in- view  GPS  receiver  and 
the  implementation  of  a  6 1  -state  tightly-coupled  Kalman  filter 
for  INS  error  calibration  provide  further  accuracy 
enhancement  and  increased  performance  in  a  jamming 
environment.  For  the  flight  demonstration.  Wide  Area  GPS 
Enhancement  (WAGE)  was  used,  which  improved  the  GPS 
signal  measurement  accuracy.  The  missile  flew  for  4-1/2  hours 
and  impacted  2.4  meters  from  the  target  center  (Figure  1 7).  A 


Figure  17.  CALCM  Precision  Strike  Demonstration 


Against  3-Meter  Target 

projected  capability  for  POGO  guidance,  based  on  GPS  IIF 
satellites  other  improvements,  and  a  guidance  accuracy 
contribution  of  1  meter,  is  2.0  meters  circular  error  probable 
(CEP). 

TARGET  SENSOR  ATTRIBUTES  FOR  PRECISION 
STRIKE  EN  THE  YE  AR  2010 

A  summary  comparison  of  sensor  alternatives  is  given  in 
Figure  1 8.  The  measures  of  merit  selected  are  sensor  per¬ 
formance  in  an  adverse  weather  environment  (cloud  cover 
and  rain),  sensor  performance  in  clutter,  ATR  discriminants, 
ATR  design  considerations,  and  weight/cost.  The  sensor 
performance  through  adverse  weather  of  clouds  and  rain 
and  operating  in  clutter  were  discussed  previously.  Fig¬ 
ure  1 8  is  based  on  the  assumptions  of  a  cloud  thickness  of 
6,000  feet,  a  cloud  base  height  of 20,000  feet,  rain  rate  of 
4  mm/hr,  and  moderate  urban  clutter.  Alternative  sensor 
capabilities  for  the  ATR  discriminants  of  angular  resolu¬ 
tion,  range  resolution,  contrast,  polarization,  temporal  (e.g., 
vibration,  motion),  multispectral,  and  gas  composition  are 
shown  in  the  figure.  The  comparison  of  other  ATR  design 
considerations  is  based  on  sensor  noise,  uniformity,  and 


Table  1.  CALCM  POGO  Demonstrated  GPS/INS  Precision  Guidance 


Error  Sources 

Standard  GPS 

POGO  Demo 

Source  of  Improvement 

User  range  error 

Satellite  ephemeris  =  4.0  m 
Satellite  clock  =  3.0  m 

5m 

URE=  2.0  m 

1.8 

1.0 

1.0 

•  WAGE  Phase  1 

•  Accuracy  improvement  initiative  (All) 

•  WAGE -complete 

•  Block  IIF  satellites 

Ionosphere 

2.3 

1.0 

•  Dual-frequency  receiver 

•  Special  algorithm  based  on  phase 

Troposphere 

2.0 

0.5 

•  Real-time  pressure  and  temperature  measurements 

•  Accurate  modeling 

Multipath 

1.2 

0.4 

•  Low  missile  multipath 

•  Phase  use  as  well  as  code 

GPS  receiver 

1.5 

0.5 

•  New  technology 

RSS  total 

6.2M 

1. 7-2.4  m 

•  URE  dependent 

LIRE  =  user  range  error 

Advanced  GPS  provides  precision  navigation  for  targeting 
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Figure  18.  Target  Sensor  Attributes  for  the  Year  2010 

latency  to  cover  a  wide  FOV.  Finally,  a  comparison  is  made 
of  weight  and  cost  of  the  alternative  sensors. 

The  best  overall  sensor  in  adverse  weather  and  clutter  is 
probably  a  SAR  sensor  operating  at  a  frequency  of  either 
the  X-band  (~10  GHz)  or  the  Ku-band  (~17  GHz).  These 
bands  permit  a  small  antenna  size  while  also  providing  a 
capability  to  penetrate  clouds  and  rain.  SAR  sensors  have 
the  flexibility  required  to  cover  an  area  search  (e.g.,  5  km 
by  5  km)  for  single-cell  target  detection,  then  switch  to 
high  resolution  (e.g.,  0.3  meter)  for  target  ID  and  targeting 
in  clutter.  SAR  sensors  can  also  provide  high-accuracy 
profiling  of  the  known  terrain  features  around  the  target 
and  derive  GPS  coordinates  of  the  target.  An  image  of  the 
high  resolution  SAR  imagery  from  the  Northrop  Grumman 
TESAR  is  shown  in  Figure  19. 


Although  today’s  SAR  sensors  have  good  imagery 
performance,  their  current  ATR  and  FAR  performance  is 
much  worse  than  the  performance  of  a  human.  As 
discussed  previously,  present  SAR  sensors  require  post¬ 
flight  human  analysis  of  'the  SAR  imagery  in  order  to 
identify/confirm  targets  and  eliminate  false  alarms.  This 
time  consuming  process  will  be  alleviated  in  the  next- 
generation  ATR,  which  will  include  other  orthogonal 
discriminants,  such  as  polarization.  The  additional 
discriminants  will  be  fused  with  the  SAR  imagery  as  well 
as  data  from  other  sensors,  to  greatly  enhance  ATR. 

Polarization  provides  a  SAR  sensor  with  the  capability  to 
extract  information  on  the  physical  shape  characteristics 
of  each  target  pixel.  Full-polarization  (transmit  left:  receive 


Figure  19.  TESAR  Image  of  U.S.  Pentagon 

left/right,  transmit  right:  receive  right/left)  algorithms  are 
under  development  by  Boeing  that  allow  complex  targets 
to  be  decomposed  into  elemental  scatterers  such  as  dihe¬ 
drals,  trihedrals,  cylinders,  helices,  and  dipoles.  Nine  dif¬ 
ferent  target  scatter  types  and  their  physical  attributes  are 
shown  in  Figure  20.  These  attributes  are  robust  over  rela¬ 
tively  wide  observation  angles.  The  application  of  full- 
polarization  algorithms  provides  more  than  10  times  the 
information  for  ATR  algorithms  than  conventional  single- 
polarization  and  significantly  improves  the  probability  of 
detection  while  reducing  FAR.  Conventional  single¬ 
polarization  SAR  algorithms  simply  exploit  imagery  based 
on  the  intensity  or  amplitude  of  the  return  signal  and 
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object  shape.  Other  discriminants  that  also  improve  the 
ATR  performance  of  SAR  sensors  include  motion  detec¬ 
tion  and  3D  imagery.  SAR  detection  of  a  moving  target 
and  estimation  of  its  velocity  is  based  on  the  temporal 
correlation  of  SAR  consecutive  images.  The  correlation 
of  doppler  in  consecutive  images  resolves  the  doppler 
ambiguity  by  associating  it  with  the  moving  target.  In  the 
obtained  sequence  the  background  is  static,  while  the 
moving  target  changes  position  from  image  to  image. 

Figure  18  also  addresses  the  strengths  and  weaknesses 
of  passive  imaging  mmW  sensors.  Passive  imaging 
mmW  sensors  have  an  advantage  over  EO  sensors  in  their 
capability  to  see  through  cloud/fog  cover.  Advantages  of 
passive  imaging  mmW  sensors  over  active  nonimaging 
radar  sensors  include  better  performance  in  clutter,  high 
contrast,  better  angular  resolution,  and  low  noise.  Figure 
2 1  illustrates  the  high  contrast  discriminant  of  passive 
imaging  mmW  sensors.  The  cold  sky  (about  35K)  is 
reflected  at  mmW  frequencies  by  the  metal  object  in  the 
image  (i.e..  Queen  Mary),  while  the  temperature  of  the 
terrestrial  clutter  and  the  fiberglass  dome  in  the  image 
is  much  higher,  about  300K.  Current  radiometers  can 
typically  detect  differences  in  temperature  of  IK. 
Radiometers  are  in  development  that  will  detect 
differences  in  temperature  of  0.  IK,  providing  a  very  high 
signal-to-noise  ratio  of  more  than  300  to  1. 

Passive  imaging  mmW  sensors  similar  to  video  cameras 
have  been  developed  by  TRW.  Array  sizes  up  to  40  x  26 
elements  have  been  produced,  with  apertures  ranging  from 


Range  to  Queen  Mary:  500  m 


TRW  94-GHz  Radiometric  Image  o«3„3_9g  ois 


Figure  21.  Passive  imaging  mmW Sensors  Provide 
High  Contrast 

0.4  to  1 .2  meters  in  diameter.  The  frequency  and  frame  rate 
demonstrated  to  date  are  89GHz  and  17  Hz  respectively. 
An  airborne  flight  demonstration  of  a  TRW  passive  imag¬ 
ing  mmW  sensor  was  successfully  conducted  during  Fall 
1997. 

A  disadvantage  of  passive  imaging  mmW  sensors  is 
relatively  low  resolution  compared  to  EO  and  SAR 
sensors.  As  noted  previously,  at  heavy  rain  rates  above  4 
mm/hr,  passive  imaging  mmW  sensors  are  highly 
attenuated.  However,  passive  imaging  mmW  technology 
is  relatively  immature.  Sensors  of  the  year  2010  time 
frame  will  have  higher  performance  and  resolution  due 
to  the  lower  noise  from  indium  phosphide  semiconductor 
substrate  technology  and  the  use  of  distributed  apertures 
as  high-resolution  interferometers.  As  an  example,  ten 
widely  spaced  apertures  could  provide  an  effective 
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aperture  that  would  be  comparable  to  the  wing  span  of  a 
UAV,  providing  more  than  an  order  of  magnitude 
improvement  in  resolution. 

Referring  back  again  to  the  Figure  1 8  assessment  of  sensor 
alternatives,  the  third  priority  sensor  for  precision  strike  is 
active  imaging  IR  based  on  LADAR.  Active  HR  has  high 
performance  in  clutter  and  provides  unique  three- 
dimensional  images  at  high  resolution  and  other 
discriminants  such  as  temporal  (e.g.,  skin  vibration,  exhaust 
gases),  exhaust  gas  composition,  and  multispectal  that 
complement  the  other  sensors  of  a  multidimensional  sensor 
suite.  Figure  22  illustrates  the  high  resolution  3D  imagery 
of  a  Lockheed  Martin  active  HR  sensor.  The  laser 
transmitter  is  usually  bore-sighted  to  a  passive  HR  sensor 
that  acquires  and  tracks  the  target.  The  capability  to  fuse 
the  target  IR  signature  with  the  reflected  laser  signature 
enhances  the  ATR  performance. 


High  Resolution  3D  Imagery 

A  technology  currently  in  development  that  will  enhance 
the  application  of  active  HR  sensors  is  tunable  lasers. 
Tunable  lasers  provide  a  multispectral  discriminant  for 
ATR. 

Active  HR  sensors  have  the  capability  to  provide  high- 
accuracy  profiting  of  the  known  features  around  the  target 
and  derive  GPS  coordinates  of  the  target. 

Target  designation  is  a  mission  that  can  be  addressed  by 
an  active  HR  sensor.  It  is  postulated  that  UAVs  of  the  year 


20 1 0  time  frame  could  carry  low  cost  laser  guided  weapons 
and  self-designate  their  targets. 

Disadvantages  of  active  HR  sensors  are  their  performance 
in  adverse  weather  and  the  latency  required  for  a  wide 
FOV  search.  Guided  submunitions  such  as  Low  Cost 
Autonomous  Attack  System  (LOCAAS)  minimize  the 
problem  by  operating  at  low  altitudes  under  clouds  and 
flying  at  low  velocity.  The  “under  the  weather”  operation 
is  effective  under  most  conditions  except  for  fog,  heavy 
rain,  and  obscurants.  Search  capability  could  be  improved 
by  adding  an  additional  wide  FOV  sensor,  such  as  passive 
imaging  IR.  The  wide  FOV  sensor  cues  the  LADAR 
sensor  for  fine  tracking  and  designation. 

Referring  back  again  to  the  Figure  18  summary 
comparison  of  sensor  alternatives,  note  that  a  passive 
imaging  IR  sensor  complements  the  baseline  SAR  sensor. 
Passive  HR  sensors  have  attributes  of  good  performance 
in  clutter,  low  noise,  multispectral  discrimination,  and 
wide  field  of  view.  The  passive  HR  multidimensional 
discriminants  (such  as  angular  resolution,  contrast,  and 
multispectral)  at  their  shorter  wavelengths  complement 
the  baseline  SAR  sensor.  There  is  a  relatively  small 
increase  in  the  sensor  suite  cost  and  weight  to  incorporate 
a  passive  HR  sensor. 

Passive  HR  focal  plane  array  (FRA)  detectors  have  good 
capability  to  sample  multiple  wavelengths,  providing  a 
multispectral  discriminant  across  a  broader  wavelength. 
Multispectral  passive  HR  sensors  have  been  produced 
with  up  to  four  colors,  based  on  either  filter  wheel  or 
stacked  FPA  technologies.  Figure  23  is  an  example  of 
the  unique  two-color  IR  signature  of  a  missile  plume  in 
the  MWIR  spectrum.  Note  that  the  average  target  intensity 
of  Color  A  (4.5  to  4.9  micron  region)  is  approximately 
seven  times  the  average  target  intensity  of  Color  B  (region 
less  than  4. 1  microns).  Also  shown  is  the  typical  terrain 
clutter  intensity  represented  by  a  300K  blackbody  and  the 
typical  solar  clutter  intensity  from  sun  glint  at  5900K.  The 
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Figure  23.  Example  of  Unique  Two-Color  IR 
Signature  of  Missile  Plume 
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Color  A/Color  B  MWIR  intensity  is  a  unique  discriminant 
of  the  missile  plume. 

EO  FPAs  cover  the  wavelength  range  from  ultraviolet  (UV) 
through  long  wave  infrared  (LWIR).  In  the  IR  wavelengths, 
a  number  of  detector  materials  are  available.  The 
production  of  most  IR  FPAs  is  based  on  mating  the 
detectors  to  a  silicon  readout  circuit,  through  indium 
columns.  This  provides  a  sandwich  or  hybrid  sensor. 
However,  there  is  a  potential  problem  from  a  mismatch 
in  the  thermal  coefficient  of  expansion,  since  high 
performance  FPAs  operate  at  cryogenic  temperatures. 
Large  IR  FPAs  tend  to  have  lower  yield  than  the  simpler, 
more  easily  produced  monolithic  silicon  based  FPAs  that 
are  used  in  visible  light  TV  cameras. 

In  the  MWIR  wavelengths  (3  to  5  microns),  the  leading 
high  performance  FPA  detectors  for  the  year  2010  time 
frame  are  InSb,  HgCdTe,  and  Quantum  Well  Infrared 
Photodetector  (QWIP)  detectors. 

InSb  and  HgCdTe  FPAs  of  640  x  480  size  are  currently 
in  production.  In  development  are  array  sizes  up  to  1024 
X  1024  with  pitch  (detector-to-detector  spacing)  as  small 
as  18  microns.  Figure  24  illustrates  the  SOTA 
advancements  of  Boeing’s  MWIR  HgCdTe  FPAs.  It  is 
postulated  that  by  the  year  2007,  HgCdTe  MWIR  FPAs 
will  be  in  production  in  a  2048  x  2048  size  with  1 0  micron 
pitch-performance,  providing  resolution  that  is  com¬ 
parable  to  that  of  the  present  SOTA  of  TV  cameras. 


QWIP  FPAs  are  based  on  stacked  thin  layers  of  materials 
that  form  quantum  wells  sensitive  to  a  broad  range  of 


Figure  24.  Example  of  Advancement  in  SOTA 
of  IR  FPAs 


wavelengths.  A  layer  is  only  a  few  molecules  thick,  creating 
energy  sub-bands  where  quantum  effects  respond  to  the 
IR  incident  radiation.  QWIP  MWIR  FPAs  have  high 
detector  uniformity  and  yield.  Disadvantages  are  relatively 
low  quantum  efficiency  and  relatively  high  dark  current. 

LWIR  (8  to  12  microns)  high  performance  tactical  FPA 
detectors  include  HgCdTe  and  QWIP.  HgCdTe  LWIR 
devices  that  are  currently  in  production  have  array  sizes 
up  to  256  X  256,  with  a  pitch  of  about  40  microns.  It  is 
projected  that  by  the  year  2000,  HgCdTe  LWIR  FPAs 
will  be  in  production  in  array  sizes  up  to  640  x  480.  The 
problem  of  producing  HgCdTe  LWIR  FPAs  in  large  array 
sizes  has  been  alleviated  by  the  use  of  a  balanced 
composite  structure.  The  balanced  composite  structure 
mitigates  the  differences  in  the  coefficient  of  thermal 
expansion  of  the  FPA.  Also  in  development  are  HgCdTe 
multispectral  FPAs  capable  of  simultaneous  detection 
in  both  the  LWIR  and  MWIR  bands. 

LWIR  FPAs  based  on  QWIP  are  currently  in  development. 
QWIP  FPAs  have  advantages  of  uniformity,  low  thermal 
stress  even  in  large  size  arrays,  and  suitability  for 
multispectral  applications.  Disadvantages  of  QWIP  LWIR 
sensors  are  a  requirement  for  cooling  down  to  less  than 
77  Kelvin,  higher  dark  current,  and  lower  quantum 
efficiency.  Cryogenic  cooler  engines  based  on  helium  are 
available  that  operate  at  temperatures  below  that  of  liquid 
nitrogen  (77  Kelvin).  However,  there  are  disadvantages  of 
shorter  lifetime,  less  temperature  control,  and  longer  cool 
down  time.  The  higher  dark  current  and  relatively  low 
quantum  efficiency  of  QWIP  FPAs  will  be  alleviated  by 
technology  advancements  for  this  relatively  immature 
technology  and  the  effective  use  of  the  multispectral 
discriminants  that  provide  enhanced  detection  at  longer 
range. 

The  other  sensor  alternatives  (active  nonimaging  radar, 
active  nonimaging  LADAR,  passive  nonimaging  IR, 
visible,  UV,  and  acoustic/seismic)  that  are  shown  in 
Figure  1 8  are  less  likely  to  be  used  in  precision  strike, 
due  to  their  combined  limitations  from  factors  such  as 
weather,  clutter,  and  angular  resolution. 

SENSOR  DATA  FUSION  ARCHITECTURE  FOR 
PRECISION  STRIKE 

As  discussed  previously,  improvements  in  the  SOTA  for 
sensor  suites  and  sensor  data  fusion  will  be  required  to 
meet  the  precision  strike  performance  requirements  for 
the  year  2010.  The  sensor  data  fusion  architecture  must 
be  sufficiently  robust  to  provide  a  high  probability  of 
selecting  the  correct  target  while  also  satisfying  the 
latency  and  cost  constraints.  As  an  example,  the 
traditional  features  for  ATR  are  based  on  2D  images  of  the 
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target,  such  as  treads  of  a  tank  target  compared  to  a 
wheeled  vehicle.  Adding  additional  target  2D  image 
features,  such  as  the  tank  gun  barrel  provides  enhanced 
confidence.  However,  greater  payoff  is  achieved  by  also 
combining  unique  features  from  multidimensional 
independent  discriminants.  Multiple  independent 
discriminants  tend  to  have  correlated  target  detection  with 
noncorrelated  false  alarms.  Thus,  a  20  percent  FAR  for  a 
single  discriminant  would  be  greatly  reduced  in 
combining  two  discriminants  (resulting  in  4  percent 
FAR),  while  a  90  percent  probability  of  detection  for  a 
single  discriminant  would  be  only  slightly  reduced  in 
combining  two  discriminants  (resulting  in  81  percent 
probability  of  detection).  Another  example  is  combining 
SAR  sensor  2D  imagery  with  the  additional  orthogonal 
feature  of  SAR  polarization. 

Orthogonal  features  in  this  context  have  noncorrelated 
false  alarms.  Orthogonal  features  often  provide  higher 
performance  with  less  required  data  bandwidth  than  the 
traditional  approach  of  using  higher  and  higher  resolution 
imagery.  By  expanding  the  number  of  orthogonal  features, 
more  difficult  targets  can  be  correctly  identified.  A 
limiting  consideration  is  the  cost  trade  of  the  alternatives 
of  (1)  increased  resolution  for  a  single  sensor,  versus 
(2)  the  fusion  of  other  orthogonal  discriminants  from 
the  same  sensor,  versus  (3)  the  fusion  of  other  orthogonal 
discriminants  from  other  sensors. 


Figure  25  is  an  example  of  sensor  data  fusion  architecture 
that  is  robust,  based  on  seven  combinations  of  classifying 
the  features  of  three  sensors.  In  the  far-left  classification 
channel,  common  features  of  sensors  A,  B,  and  C  are 
classified.  In  the  case  of  a  sensor  suite  consisting  of  three 
imaging  sensors  in  different  wave  bands,  the  target  pixels 
of  the  bore-sighted  sensors  would  be  registered  to  obtain 
a  common  image  in  each  wave  band.  The  unique  features 
of  the  target  (e.g.,  tank  treads,  tank  gun  barrel)  are  classified 
for  common  features.  Other  common  features,  such  as 
polarization  at  different  wavelengths,  are  also  classified. 
The  advantage  of  the  serial  configuration  of  the  far-left 
classification  channel  is  that  false  alarms  and  decoys  are 
likely  to  be  rejected.  The  common  features  must  be 
measured  in  all  three  wave  bands  to  be  accepted  as  a 
target.  A  disadvantage  of  the  serial  fusion  in  the  far-left 
channel  is  lower  detection  probability,  particularly  for  low 
signature  targets  in  clutter.  The  next  three  classification 
channels  provide  combinations  of  the  common  features 
from  two  sensors.  These  channels  tend  to  have  higher 
detection  but  also  higher  false  alarms.  Finally,  the  three 
classification  channels  on  the  far  right  individually  classify 
the  features  of  the  individual  sensors  A,  B,  and  C.  The 
classification  is  based  on  the  data  fusion  of  discriminants 
(e.g.,  image  features,  polarization)  for  each  individual 
sensor.  These  chaimels  tend  to  have  the  highest  detection 
but  may  also  have  the  highest  false  alarms.  Above  the 
seven  classification  channels  is  the  decision  level  fusion. 


Low  FAR 
Lower  Detection 


Moderate  FAR 
Moderate  Detection 


High  FAR 
Higher  Detection 
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Figure  25.  Example  of  Robust  Sensor  Data  Fusion  Architecture  for  Low  FAR  and  High  Detection  ATR 
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The  decision  level  fusion  processes  the  results  of  the 
seven  individual  channel  classifications  to  yield  a  fused, 
optimized  target  identification  decision.  Decision  level 
fusion  of  multiple  sensors  is  a  relatively  immature 
technology,  and  is  currently  more  of  an  art  than  a  science. 

Figure  26  is  an  example  of  sensor  data  fusion  based  on  an 
allocation  of  discriminants  for  ATR.  The  example  is  based 
on  three  sensors  -  Sensor  A:  SAR,  Sensor  B:  passive 
imaging  mmW,  and  Sensor  C:  active  imaging  IR.  Two- 
dimensional  imagery  from  an  angular  resolution 
discriminant  was  selected  as  a  common  discriminant  for 
all  three  sensors.  This  information  could  be  used  in  the 
far-left  classification  channel  of  Figure  25.  Requiring  a 
classification  of  target  image  features  (e.g.,  tank  treads, 
tank  gun  barrel)  for  three  different  wavelengths  (e.g.,  X- 
band,  mmW,  IR)  mitigates  the  problem  of  false  alarms. 
An  example  of  the  other  classification  channels  in  Figure 
25  is  the  assumed  common  discriminants  of  Sensors  A 
and  B  (2D  image,  temporal),  Sensors  A  and  C  (2D  image, 
polarization).  Sensors  B  and  C  (2D  image,  contrast),  as 
well  as  the  individual  discriminants  of  Sensors  A,  B, 
and  C. 


Figure  26.  Example  of  ATR  Discriminant  Allocation  for 
Sensor  Data  Fusion 


Figure  27  illustrates  how  sensor  data  fusion  provides 
higher  confidence  to  discriminate  and  identify  the  target, 
over  the  traditional  approach  of  using  only  image 
resolution.  The  traditional  approach  to  ATR  of  a  target  in 
clutter  is  based  on  statistical  pattern  recognition  of  the 


2D  image  features  such  as  height-to-width  ratio,  unique 
components  (e.g.,  tank  treads,  tank  gun  barrel),  and  contrast 
(Intensity  of  the  target  resolution  cells  [pixels]  compared 
to  the  background).  An  ATR  task  (e.g.,  detection, 
classification,  recognition,  etc.)  can  be  related  to  the 
number  of  sensor  resolution  cells  or  line  pairs  across  a 
maximum  dimension  of  the  target.  A  line  pair  is  the 
instantaneous  field  of  view  (IFOV)  resolution  of  the  sensor. 
NxIFOV  is  a  maximum  dimension  (e.g.,  width,  length)  of 
the  target.  For  a  square  target,  there  are  resolution  cells 
imaged  by  the  sensor.  In  the  case  of  detection,  2-4  line 
pairs  are  required  to  indicate  that  an  object  is  present  in 
the  clutter.  In  the  case  of  classification,  3-6  line  pairs  are 
required  to  classify  fhe  target  in  a  broad  class  of  object 
types,  such  as  tracked  vehicles.  In  the  case  of  recognition, 
6-14  line  pairs  are  the  traditional  criteria  to  put  the  object 
into  a  subclass,  such  as  a  tank.  In  the  case  of  identification, 
8-22  line  pairs  are  the  traditional  criteria  for  a  specific  object 
type  within  a  class  of  objects,  such  as  a  T-72  tank.  Finally, 
discrimination  involves  distinguishing  real  targets  (e.g.,  a 
real  T-72  tank)  from  decoys  (e.g.,  a  replica  of  a  T-72  tank). 
A  problem  with  single  sensor  monochrome  imagery  is  the 
point  of  diminishing  return  where  the  confidence  in  ATR 
does  not  improve  with  increased  resolution.  In  the  example 
of  Figure  27,  a  SAR  sensor  could  avoid  the  problem  by 
fusing  2D  imagery,  polarization,  and  multispectral 
discriminants  for  enhanced  ATR  performance.  Sensor  data 
fusion  not  only  provides  higher  confidence  in  each  task, 
but  also  provides  a  capability  for  the  more  difficult  ATR 
tasks,  such  as  discrimination,  that  may  not  be  achievable 
using  traditional  imaging-only  ATR. 

The  previous  discussion  has  focused  on  sensor  level  and 
decision  level  fusion.  A  potential  disadvantage  is  the 
discarded  information  associated  with  the  same  pixel 
data  correlations  between  sensors.  However,  the 
alternative  of  pixel  level  fusion  at  the  raw  data  level  may 
also  have  problems,  due  to  differences  in  pixel  size, 
resolution,  field-of-view,  and  boresighting.  The  issue  of 
sensor-level  versus  pixel-level  fusion  is  an  on-going  debate 
in  the  sensor  data  fusion  community. 

CONCLUDINGREMARKS 

Sensor  data  fusion  technologies  will  have  high  payoff 
for  precision  strike  in  the  year  2010.  Benefits  include 
reduced  workload  for  the  image  analyst,  mission  planner, 
and  pilot;  improved  safety  for  the  pilot  and  friendly 
forces/civilians;  higher  lethality;  and  the  potential  to 
reduce  the  cost  per  target  kill.  There  is  no  doubt  that 
sensor  data  fusion  provides  higher  performance  over 
traditional  2D  imaging  ATR  that  uses  a  single  wavelength 
from  a  single  sensor. 


Figure  27.  Example  of  Benefit  of  Sensor  Data  Fusion 


ATR  development  should  consider  sensor  alternatives 
that  capture  the  most  important  target  features  and 
efficiently  run  the  ATR  algorithms.  Similarly,  the  sensor 
development  should  consider  ATR  algorithm  alternatives 
that  exploit  sensor  performance  and  capture  the  most 
important  target  features.  A  problem  is  that  the  ATR 
community  is  typically  funded  independently  from  the 
sensor  community  and  there  is  little  incentive  to  work 
together.  Unless  the  ATR  and  sensor  communities 
communicate  more  on  system  level  trades,  the  system  level 
result  will  be  nonoptimum  (Figure  28). 


Selection  of  the  best  approach  for  sensor  data  fusion 
requires  an  assessment  of  alternative  concepts  and 


Figure  28.  Sensors  and  ATR  Need  to  be  Developed 
in  Concert 


technologies  to  provide  a  traceable  path  from  system- 
level  requirements  to  technology  requirements.  However, 
developers  of  sensors  and  ATR  algorithms  usually  have  a 
conflict  of  interest  in  conducting  technology  assessments. 
The  sensor  community  may  advocate  sensors  that  are 
based  more  on  what  they  wish  to  sell  and  not  by  reasoned 
consideration  of  enhanced  ATR.  Similarly,  the  ATR 
community  may  advocate  favorite  algorithms  that  are  not 
optimized  for  sensors.  The  assessment  of  alternatives  for 
sensor  data  fusion  concepts  and  technologies  should  be 
conducted  by  an  “honest  broker”  that  is  not  biased  by 
prior  Investments  in  either  sensors  or  ATR.  Figure  29 
illustrates  a  study  to  evaluate  alternatives  for  sensor  data 
fusion  concepts  and  technologies.  The  study  duration  is 
typically  3  to  6  months  and  the  scope  is  approximately  2  to 
4  person-years.  Typical  study  tasks  are  mission  analysis/ 
scenario  definition;  sensor  data  fusion  requirements,  trade 
studies  and  sensitivity  analysis;  assessment  of  the 
physical  and  avionics  Integration  of  the  sensor  on  the 
aircraft/UAV/weapon  platform;  sensor  data  fusion  concept 
design  synthesis;  and  technology  assessment  and 
development  roadmap.  The  recommended  system 
concepts  and  technologies  are  a  system-level 
“requirements  pull”  for  sensor  fusion  technologies  that 
balances  the  “technology  push”  advocacy  from  the  sensor 
and  ATR  communities. 
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•  Mission/Scenario  Definition 


•  Sensor  Data  Fusion 
Requirements,  Trade 
Studies  and  Sensitivity 
Analysis 


•  Physical  Integration  of 
Platform(s)  and  Sensors 


•  Sensor  Data  Fusion 
Concept  Design  Synthesis 

•  Technology  Assessment  and 
Development  Roadmap 


Figure  29.  Example  of  an  Assessment  of  Alternative  Sensor  Data  Fusion  Concepts  and  Technologies 


QUOTATIONS 

•  “When  I  see  a  bird  that  walks  like  a  duck  and  swims 
like  a  duck  and  quacks  like  a  duck,  1  call  that  bird  a 
duck.” 

~  Richard  C.  Cushing  ~ 

•  “A  pile  of  facts  is  no  more  a  science  than  a  pile  of 
bricks  is  a  house.” 

~  J.  Henri  Poincare  ~ 

•  “Where  is  the  wisdom  we  have  lost  in  knowledge? 
Where  is  the  knowledge  we  have  lost  in  information?” 

~  T.S.  Elliot  ~ 

•  “He  can  run  but  he  can’t  hide.” 

~  Joe  Louis  ~ 
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Abstract 

Modern  cockpit  environments,  covering  highly  integrated  and 
complex  automatic  functions,  pose  various  demands  on  the 
crew.  In  unusual  situations  the  crew  often  is  overtaxed  and  acts 
erroneously.  “Clumsy  automation”  is  considered  to  be  a  major 
reason  for  deficiencies  concerning  the  interaction  between 
cockpit  crew  and  aircraft  systems.  Cognitive  systems  appear  to 
be  a  promising  approach  to  overcome  these  deficiencies  in 
future  cockpits.  They  are  being  developed  covering  human-like 
capabilities  in  the  interpretation  and  diagnosis  of  the  situation, 
planning  and  decision  making  and  the  execution  of  a  plan.  In 
this  paper  a  general  survey  on  the  principals  of  cognitive 
cockpit  assistance  will  be  given.  Demands  and  requirements  for 
an  appropriate  automation  and  a  generic  functional  structure  of 
a  cognitive  assistant  system  will  be  introduced.  A  prototype 
system,  CAMA,  the  Crew  Assistant  Military  Aircraft,  its 
capabilities  and  functional  units  (modules)  are  presented  and 
described  in  detail.  In  future  combat  transport  aircraft, 
constraints  created  by  low  level  flying  in  a  high  risk  theater,  the 
high  rate  of  change  of  information  and  short  reaction  times 
required  will  produce  physiological  and  cognitive  problems  for 
pilots.  CAMA  is  designed  taking  into  consideration  the 
approach  of  human-centered  automation. 

Introduction 

Automation  technology  in  the  past  improved  significantly 
flight-safety,  mission-effectiveness  (satisfying  multiple  goals) 
and  efficiency  (e.g.  fuel  economy,  timelines).  However,  recent 
accident  investigations  reveal  that  the  human  operator  might 
become  overcharged  by  the  current  concept  of  cockpit 
automation. 

The  modem  cockpit  is  a  multitask  environment,  comprising 
four  task  categories: 

•  to  aviate, 

•  to  navigate, 

•  to  communicate  and 

•  to  manage  systems. 

Executing  these  tasks,  the  crew  is  permanently  trying  to  comply 
with  a  number  of  concurrent,  sometimes  competing,  often 


conflicting  objectives.  Multiple  actions  have  to  be  performed  to 
accomplish  them.  The  allocation  of  operator  tasks  to  automatic 
devices  is  often  seen  as  the  appropriate  design  strategy  to 
reduce  workload.  However,  the  crew  remains  still  responsible 
for  operating  the  aircraft.  Complexity  and  autonomy  of 
automation,  the  coupling  between  or  among  automated 
functions  and  the  inadequate  feedback  have  been  identified  as 
leaks  in  the  system's  concept  [1].  Monitoring  and  supervising 
requirements  become  more  prevailing  and  that  may  increase  the 
pilot's  workload  in  unusual  situations.  Managing  the  automatic 
devices  and  being  aware  of  their  behavioral  features  becomes 
more  and  more  difficult. 

Automation  increases  the  aircraft's  performance  and  by  that  its 
mission  capabilities.  The  complexity  of  planning  and  decision¬ 
making  tasks  increases  correspondingly  and  takes  pilot's  mental 
capacity.  Aircraft  manufacturers  and  flying  organizations  try  to 
reduce  crew  decision-making  as  much  as  possible.  This  is  done 
by  means  of  automation  and  by  establishing  standard 
procedures  and  checklists  to  cover  anticipated  failures  or 
emergencies  [2]  [3].  Natural  decision  making  environments  can 
be  turbulent,  with  time-varying  goals  and  requirements.  Pilot's 
tasks  alter  with  increased  automation  as  to  decreasing  physical 
activities  and  increasing  cognitive  activities.  The  human  being 
is  well  equipped  with  cognitive  capabilities  concerning  his 
natural  tasks.  However,  flight  guidance  provides  quite  different 
challenges.  The  crews  have  to  make  many  different  kinds  of 
decisions,  but  all  involve  situation  assessment,  choice  among 
alternatives,  including  risk  assessment.  Effective  crews  are 
characterized  by  [2]: 

•  Good  situation  awareness. 

•  High  levels  of  metacognition. 

•  Shared  mental  models  (based  on  explicit 
communication). 

•  Efficient  resource  management. 

Assisting  the  pilot  in  his  flight  guidance  task  by  a  technical 
system  should  compensate  humans  deficencies  in  the  cognitive 
process  and  improve  decision-making  in  any  situation. 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element", 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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Cognitive  Assistant  Systems 


The  human  information  processing  comprises 

•  Monitoring  of  the  Situation, 

•  Diagnosis  of  the  Situation, 

•  Planing  and  Decision  Making  and 

•  Plan  Execution/Activation. 


Situation  monitoring  comprises  the  determination  of  situational 
attributes,  based  on  the  perception  of  the  relevant  objects.  The 
situation  diagnosis  completes  the  picture  of  the  situation  by 
evaluation  of  the  situational  attributes  against  the  governing 
objectives,  plans  and  actions,  thereby  identifying  problems. 
Conflict  resolution  is  performed  by  planning  and  decision 
making.  Planning  comprises  the  generation  of  alternatives  for 
interim-goals,  plans,  tasks  and  actions.  Selection  and 
instantiation  of  the  most  appropriate  alternative  subject  to  the 
governing  goals  is  performed  within  the  decision  making 
process  which  is  entailed  by  the  execution  of  the  plan. 


Rassmussen  [3]  distinguishes  human's  cognitive  capabilities  by 
the  three  levels  of  knowledge-based,  rule-based  and  skill-based 
behavior  (see  Figure  1) 
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Figure  1 :  Human's  Level  of  Cognitive  Behavior  [3] 

The  cognitive  process  passes  all  levels  of  human-behavior. 
Situation  interpretation  starts  on  a  skill-based  level  by  the 
extraction  of  features.  The  diagnosis  of  the  situation,  with 
regard  to  identification  and  resolution  of  conflicts  can  become  a 
knowledge-based  process.  Plans  and  tasks  are  executed  in  the 
rule-based  level  resulting  in  various  actions. 

Current  automatic  devices  provide  good  performance  within 
their  specified  restricted  functional  scope.  The  representation  of 
the  situation  within  these  devices  is  restricted  to,  consequently. 
This  provides  fundamental  implications  and  resulting  problems 
[4]: 

•  Automation  performs  tasks  without  knowledge  of  the 
complete  situation,  current  goals,  plans  and  further  parallel 
tasks. 

•  The  assessment  of  the  situation  remains  by  the  pilot,  who 
has  to  determine  wether  the  current  situational  scope  fits  the 
specification  of  the  activated  automatic  device. 

•  The  pilot  has  to  ensure  that  all  automatic  devices  are 
supporting  the  current  goals  the  pilot  tries  to  comply  with. 


This  is  the  reason  why  increasing  of  cockpit  automation 
complexity,  managing  the  systems  becomes  more  and  more 
prominent. 

Cognitive  assistant  systems  cover  all  of  the  human-like 
capabilities  in  Figure  1,  using  the  representation  of  the 
complete  picture  of  the  situation  over  all  levels  of  behavior  (see 

Figure  2). 

These  systems  work  in  parallel  to  the  pilot  as  to  assessment  of 
the  situation  and  resolution  of  conflicts  and  support  mutual 
assistance.  The  underlying  concept  for  the  interaction  between 
assistant  and  pilot  is  the  human-centered  approach,  with  the 
following  fundamental  design  criteria  [1][5]: 

•  Human  operators  must  be  actively  involved. 

•  Human  operators  must  be  adequately  informed. 

•  Human  operators  must  be  able  to  monitor  the  automation 
assisting  them. 

•  The  automated  system  must  also  monitor  the  human 
operator. 

•  Automated  systems  must  be  predictable. 

•  Every  intelligent  system  element  must  know  the  intent  of 
other  intelligent  systems  elements. 


Figure  2:  Conventional  vs  Cognitive  Automation 

Onken  [6]  stated  two  basic  functional  requirements  for  the 
behavior  of  a  cognitive  assistant  system,  which  are  in  line  with 
the  design  criteria  of  the  human-centered  approach: 

(1)  "Within  the  presentation  of  the  full  picture  of  the  flight 
situation  it  must  be  ensured  that  the  attention  of  the  cockpit 
crew  is  guided  towards  the  objectively  most  urgent  task  or 
sub-task  of  that  situation." 

(2)  "If  basic  requirement  (1)  is  met,  and  if  there  still  comes  up  a 
situation  with  overcharge  of  the  cockpit  crew  (in  planning 
and  execution),  then  this  situation  has  to  be  transfered  -  by 
use  of  technical  means  -  into  a  situation  which  can  be 
handled  by  the  crew  in  a  normal  manner." 

These  requirements  lead  to  the  functional  structure  of  the 
cognitive  assistant  system  as  depicted  in  Figure  3.  For  a 
comprehensive  understanding  of  the  situation,  the  process  of 
situation  analysis  comprises  both  perception  of  objects  and 
pertinent  features  in  a  number  of  levels  of  abstraction  as  well  as 
the  process  of  situation  diagnosis  which  based  on  the 
perception,  recognizes  and  predicts  conflicts,  caused  by  events 
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in  the  domain  of  either  aircraft,  pilot  or  environment.  This 
closely  resembles  the  human  way  of  situation  analysis 

In  order  to  recognize  potential  opportunities  in  addition  to 
conflicts  alternatives  are  generated  for  goals,  plans,  tasks  and 
actions,  including  that  one,  which  represents  the  given 
flightplan.  Only  the  conflict-free  alternatives  are  passed  to  the 
'Decision  Finder  of  How  to  Proceed'.  The  conflict  solving 
process  is  ranking  these  alternatives  on  the  basis  of  the  mission 
success  criteria. 

The  Dialogue  Manager  insures  effective  communication  with 
the  crew.  This  is  the  front-end  of  an  assistant  system.  It  presents 
all  necessary  and  useful  information  in  a  way,  that  is  easy  to 
comprehend.  Messages  to  the  cockpit  crew  should  be  tuned  and 
tailored  to  the  current  situation,  especially  with  respect  to 
minimize  the  demands  on  crew  resources.  Inputs  to  the  system 
should  allow  initialization  of  services  and  decision  support 
without  tedious  or  distracting  input  actions. 


Figure  3:  Structure  of  a  Cognitive  Assistant  System. 

Knowledge  processing  needs  a  dynamic  object-orientated 
representation  of  the  situation-describing  objects.  The 
representation  covers  sensor  data  on  the  lowest  level  as  well  as 
very  abstract  objects  like  a  whole  flight  plan  or  for  instance  the 
recognized  intent  of  the  crew. 

Other  knowledge  bases  are  needed  to  express  and  enable  access 
to  domain  knowledge  and  to  permit  inferencing.  Models  about 
motives  and  goals,  task  selection,  execution  knowledge  and 
demand  for  resources  as  well  as  behavior  models  are  important 


examples  of  this  kind  of  knowledge,  executed  by  additional 
information  about  the  crew  member. 

Static  data  bases  for  navigation  purposes  or  threat  data  bases 
can  already  be  considered  as  standard,  keeping  in  mind,  though, 
that  incompletness,  incorrectness  and  aging  effects  have  to  be 
coped  with. 

The  expert  knowledge  embodied  in  the  system  has  to  be 
obtained  in  a  systematic  way. 

Knowledge  acquisition  appears  as  a  potential  bottle  neck  during 
development  of  the  knowledge-based  assistant  system.  Well- 
defined  and  efficient  algorithms  and  methods  have  to  be  used  to 
cope  with  the  ill-structured  and  uncertain  real  world. 

In  order  to  increase  user  acceptance,  it  is  desirable  that  the 
system  contains  a  justification  or  explanation  component.  First 
of  all  the  user  should  be  conscious  of  the  rules  that  are  applied 
in  the  algorithm  to  obtain  a  solution  or  system  state.  This  will 
provide  confidence  in  the  system  outputs. 

System  self-diagnosis  makes  sure  that  the  hints  and  services  to 
the  crew  will  be  really  useful.  The  system  must  be  able  to 
realize,  if  information  concerning  the  actual  situation  might  be 
insufficient  to  assist  the  crew,  or  that  the  system  itself  is  not 
working  all  right  and  needs  to  be  corrected. 

CAMA  -  Crew  Assistant  Military  Aircraft 

In  future  combat  transport  aircraft,  low  level  flying  in  a  high 
risk  theater,  the  high  rate  of  change  of  information  and  short 
reaction  times  required  will  create  physiological  and  cognitive 
problems  for  pilots.  From  the  cognitive  point  of  view,  low  level 
flying  over  rapidly  changing  terrain  elevation  coupled  with 
complex  and  dynamic  tactical  environment  will  result  primarily 
in  difficulties  to  maintain  situation  awareness.  It  still  seems 
impossible  to  ensure  the  pilot’s  situation  awareness  as  the 
dominating  requirement  for  high  level  mission  performance  and 
safety. 

CAMA,  the  Crew  Assistant  Military  Aircraft,  is  a  cognitive 
cockpit  assistant  system  (see  also  [7]  [8]).  It  is  being  developed 
and  tested  in  close  cooperation  between  the  Daimler-Benz 
Aerospace  AG  Ottobrunn,  Deutsches  Zentrum  fur  Luft-  und 
Raumfahrt  Braunschweig,  Elektronik-  und  Logistiksysteme 
GmbH  Miinchen  and  Universitat  der  Bundeswehr  Milnchen. 
The  system  is  divided  into  several  functional  units  (modules), 
each  supporting  the  execution  of  a  specific  task  category  (see 
Figure  4). 

Interpretation  of  the  Situation 

The  perception  and  determination  of  situational  objects  and 
features  comprises  the  following  modules: 

The  Environment  Interpreter  (El)  provides  information 
concerning  the  actual  weather  and  the  technical  environment. 
Thunderstorms,  areas  of  turbulence,  the  atmospheric  conditions 
as  well  as  objects  of  special  interest  (e.g.  airborne  vehicle)  are 
recognized  and  located  relative  to  the  aircraft's  current  position. 

The  airspace  surveillance  task  is  performed  by  the  Traffic  Alert 
and  Collision  Avoidance  System  (TCAS).  In  the  case  of  a 
potential  traffic  conflict,  four  levels  of  information  and 
advisories  are  issued  respectively.  Visual  and  acoustic  advises 
and  recommendations  are  given.  CAMA  monitors  and  supports 
the  execution  of  the  conflict  resolution. 
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Ground  proximity  is  detected  by  the  Terrain  Interpreter  (TI).  A 
hyperplane  of  possible  flight  trajectories,  achievable  by  full 
exploitation  of  the  aircraft's  performance  capabilities  is 
calculated  and  checked  for  intersections  with  the  terrain,  using  a 
terrain  database. 

The  aircraft's  state  is  monitored  by  the  System  Interpreter  (SI) 
[9].  In  the  case  of  a  detected  malfunction  the  module  initiates  a 
system  diagnosis  to  resolve  the  reason  for  the  failure  occurred. 

The  module  Computer  Vision  External  (CVE)  [10]  serves  the 
interpretation  of  the  surrounding  environment  using  its 
computer  vision  capabilities.  Determination  of  aircraft's  actual 
state  in  respect  to  the  environment.  The  module  supports 
approach  and  landing  even  on  unequipped  airfields.  In  addition 
obstacles  on  the  runway  are  recognized. 

The  Tactical  Situation  Interpreter  (TSI)  [1 1]  computes  a  threat 
map.  A  digital  terrain  elevation  map  and  the  present  tactical 
scenario  (e.g.  hostile  SAM  stations)  are  taken  into 
consideration.  The  threat  model  comprises  the  maximum  range, 
operationability,  efficiency  along  range  and  respective  models 
for  threat  area  overlapping. 

The  module  Computer  Vision  Internal  (CVI)  [12]  provides 
information  concerning  the  pilot's  point  of  gaze.  A  bifocal 
camera  is  installed  in  the  cockpit's  front-panel  for  this  purpose. 
The  information,  for  instance  the  moving  line  of  sight  to  a 
control  device  or  to  a  certain  indicator,  could  be  used  to 
confirm  the  need  for  a  warning  or  a  hint.  This  information  can 
also  lead  to  a  better  assessment  of  the  pilot’s  resources  [13] 


Interpretation  of  Pilot  Behavior 

Monitoring  of  the  pilot  flying  by  a  pilot  non-flying  in  the  multi¬ 
crew  cockpit  is  one  of  the  reasons  that  air  transport  is  as  safe  as 
we  are  used  to.  In  the  same  way  pilots  monitor  the  actions  of  air 
traffic  controllers,  and  in  the  turn,  controllers  monitor  the 
behavior  of  the  aircraft  they  control.  The  assistant  system 
should  have  the  same  kind  of  capability. 

With  specific  attention  to  potential  failure  points  (e.g.  those 
documented  in  aviation  operations)  and  by  the  application  of 
information  technology  and  the  support  of  the  various  aircraft 
systems  more  comprehensive  and  effective  monitoring  of  both 
pilots  and  controllers  is  applicable.  Within  the  human-centered 
approach  automation  can  and  should  be  thought  of  as  a  primary 
monitor  of  human  behavior  in  exactly  the  same  way  that 
humans  are  the  primary  monitors  of  machine  behavior  [1]. 

The  Pilot  Behavior  Interpretation  (FBI)  generates  the  expected 
pilot  behavior  in  two  ways  depending  on  the  behavioral  pilot 
model  used.  A  normative  model  [14]  describes  deterministic 
pilot  behavior  as  documented  in  pilot  handbooks  and  air  traffic 
regulations.  Primarily  rule-based  behavior  is  considered.  The 
modeling  is  done  for  all  flight  segments  (taxi,  takeoff, 
departure,  IFR-cruise,  tactical  flight,  drop,  approach,  landing) 
and  the  following  tasks: 

a)  situation  analysis 

•  Recognition  of  actual  flight  segment. 

•  Recognition  of  process  of  plan  execution  related 
to  flight  plan  procedures. 
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b)  pilot  actions  /  performance 

•  Primary  flight  guidance  (altitude,  course, 
airspeed,  power  setting,  climb/descent  rate,  pitch 
attitude). 

•  Operation  of  flaps,  speed  brakes  and  gear. 

•  Drop  procedure. 

•  Operation  of  ramp. 

•  Radio  navigation. 

An  adaptive  model  [15]  covers  behavioral  parameters  of  the 
individual  pilot,  when  specifically  differing  from  the  normative 
model. 

The  expected  pilot  behavior  as  provided  by  the  PBI  is 
compared  with  the  pilot's  actual  behavior  by  the  Pilot  Intent 
and  Error  Recognition  (PIER)  [16]  [17].  In  case  of  detected 
deviations  the  module  PIER  tries  to  figure  out,  wether  the 
deviation  was  initiated  erroneously  or  intentionally.  Detected 
errors  are  issued  to  the  Flight  Situation  and  Threat  Interpreter. 
Error  detection  will  help  the  pilot  to  correct  slips  and  to  focus 
his  attention  on  most  important  or  critical  events. 

By  monitoring  pilot  actions  as  well  as  the  mission  context,  the 
system  is  able  to  compare  the  pilot’s  action  to  a  set  of  behavior 
hypotheses.  In  case  of  an  intentional  deviation  from  the  flight 
plan,  the  module  checks,  if  the  behavior  fits  to  one  set  of  a 
given  set  of  intent  hypotheses. 

These  hypotheses  represent  behavior  patterns  of  pilots,  for 
example,  when  commencing  a  missed  approach  or  avoiding  a 
thunderstorm.  With  the  intention  recognized,  support  like  re¬ 
planning  is  initiated,  and  the  overall  loop  could  be  closed 
without  further  inputs  of  the  pilot. 

In  the  more  tightly  integrated  system  of  the  future,  such  cross¬ 
monitoring  between  man  and  machine  and  vice  versa  will  be 
the  key  to  improved  system  safety. 

Situation  Diagnosis 

Mission-oriented  goals  are  derived  from  the  mission  order.  The 
mission  order  comprises  instructions  and  constraints,  which  are 
to  be  kept  (e.g.  entrance-corridors  to  gaming  area,  drop-point, 
TOT,  etc.).  The  recognition  and  prediction  of  potential  conflicts 
and  the  initiation  of  suitable  measures  for  conflict  resolution  are 
the  primary  tasks  of  the  Flight  Situation  and  Threat  Interpreter 
(FTI). 

The  FTI  initiates  the  module  Mission  Planner  (MP)  to  generate 
a  complete,  conflict-free  flightplan.  If  the  mission  order  leads 
the  aircraft  into  an  area  with  hostile  radar  coverage,  the  Low 
Altitude  Flight  Planner  [10]  is  started  additionally  for  the 
calculation  of  a  low  altitude  flightplan  as  well  as  the  pertinent 
trajectory.  FTI  controls  the  planning  parameters.  They  are 
provided  to  the  planning  modules  and  comprise  origin  and 
destination,  corridors  to  be  planned  through,  civil  and  tactical 
waypoints  as  well  as  detailed  drop  procedures. 

Crew  constraints,  e.g.  individual  route  preferences  are  included 
as  well.  Generated  routes  are  proposed  to  the  cockpit  crew,  and 
are  accepted,  modified  or  refused  respectively. 

Erroneous  crew  intents  and  other  crew  errors,  recognized  by  the 
module  PIER  are  monitored  and  hints  and  warnings  to  the  crew 
will  be  initiated.  Sensible  deviations  from  the  flight  plan. 


intentionally  carried  out  by  the  crew  are  incorporated  in  the 
actual  flightplan. 

Monitoring  of  the  flightplan  and  evaluation  against  the  situation 
is  done  permanently.  Conflicts  within  the  planned  route,  e.g. 
new  threats  popping  up  within  the  operation  area  or  corridors, 
weather  deterioration  en-route  or  at  the  destination,  etc.,  will  be 
recognized  and  suitable  mission  planning  routines  will  be 
triggered. 

Mission  Planning 

CAMA  comprises  two  flight  planning  agents,  serving  the 
knowledge-based  conflict  resolution  task.  Both  are  coordinated 
and  controlled  by  the  FTI  via  a  set  of  various  parameters. 

The  Mission-Planner  (MP)  [18]  calculates  a  flightplan  for  the 
purpose  of  a  civil  IFR  flight.  The  module  creates  and  maintains 
a  'take  off -to-landing  mission  flight  plan,  including  routes, 
profiles,  time  estimates  and  fuel-planning,  taking  into 
consideration  the  mission  constraints,  gaming  area,  ATC 
instructions,  aircraft  state  (e.g.  fuel  contingency,  system 
failures),  environmental  data,  etc.  Two  lateral  route  alternatives 
-  a  standard  IFR  routing  as  well  as  a  radio-navigation  route  - 
are  calculated  and  proposed  to  the  crew.  Acceptance, 
modification  and  rejection,  all  of  those  pilot  reactions  are 
possible. 

The  task  of  the  Low-Altitude  Flight  Planner  (LAP)  [11]  is  the 
calculation  of  a  3D  trajectory  with  a  maximum  probability  of 
survival  in  a  hostile  environment.  This  is  achieved  by  avoiding 
threatened  areas  if  possible,  minimizing  the  exposure  to 
unknown  threats  and  keeping  clear  of  the  terrain.  Therefore,  the 
planning  constraints,  the  tactical  elements  and  the  resulting 
threat  map,  the  terrain  elevation  data  and  the  aircraft 
performance  data  are  taken  into  consideration. 

Crew  Interface 

Communication  between  CAMA  and  the  cockpit  crew  plays  an 
important  role.  The  kind  of  information  to  be  transmitted  in 
either  direction  varies  with  respect  to  the  different  situations 
and  demanded  tasks.  The  information  flow  from  CAMA  to  the 
crew  and  vice  versa  is  controlled  exclusively  by  the  module 
Dialogue  Manager  (DM)  [13][19]. 

Pilot's  inputs  are; 

•  Requests  of  new  flightplan  proposals. 

•  Activation  or  rejection  of  proposal. 

•  Activation  of  actions  related  to  warnings. 

•  Retrieval  of  information  (e.g.  ATIS,  navigational 
aids,  reconnaissance  information,  mission 
progress). 

•  Autopilot  operations. 

•  Configuration  of  the  man/machine  interface  (e.g. 
heading-up,  range,  display  modes). 

CAMA  outputs  are: 

•  Presentation  of  calculated  flightplan  proposals. 

•  Situation  presentation  (e.g.  weather,  tactical 
situation,  terrain,  airports) 

•  Warnings  about  detected  conflicts. 
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•  Recommendation  of  explicit  actions. 

•  Messages  in  reply  to  requests. 

•  Acknowledgment  of  speech-input. 

•  Presentation  of  complex  actions  (e.g.  approach 
briefing) 

The  Dialogue  Manager  coordinates  the  following  interface 
components: 


•  Moving  map  display  (see  Figure  5)  providing 
various  information  and  map-layers  (e.g.  low 
level  flying  chart,  terrain,  calculated  threat). 

•  Secondary  display  for  e.g.  flight-log,  ATIS, 
horizontal  scenario  overview,  briefings. 

•  Speech  output  systems. 


•  Context  sensitive  speech  recognition  system. 


Figure  6:  Primary  Flight  Display  in  3D-mode. 


Additional  man-machine-interface  devices  provide  support  for 
the  flight-guidance  task.  For  flying  in  low  altitudes  under 
demand  of  visual  flight  rules,  a  primary  flight  display  (see 
Figure  6)  provides  a  3-dimensional  presentation  of  the 
surrounding  environment.  The  calculated  trajectory  is  presented 
as  a  tunnel. 


Flight  Simulator  Trials  and  In-Flight  Experiments 

CAMA  is  integrated  in  the  flight  simulator  of  the  Universitat 
der  Bundewehr  MUnchen  (University  of  the  German  Armed 
Forces  Munich).  In  November  '97  and  May  '98  flight  simulator 
test  runs  were  conducted  with  10  military  transport  pilots.  Each 
pilot  conducted  two  tactical  missions  of  1  hour.  The  results  of 
these  flight  simulator  tests  are  also  presented  at  this  symposium 
[20]. 

Currently  CAMA  is  being  integrated  in  the  in-flight  simulator 
ATTAS  of  the  DLR  (German  Aerospace  Research 
Establishment  Braunschweig)  and  will  be  demonstrated  in 
flight  experiments  scheduled  for  the  early  year  2000. 

Further  Research 

At  this  point,  interactive  planning  in  terms  of  dealing  with 
specific  pilot  request  at  any  time,  is  not  developed  yet  to  an 
extent  which  will  be  desirable  in  an  operational  system. 
Although  there  is  no  principal  difficulty  to  achieve  this  on  the 
basis  of  the  CAMA  system  structure,  it  seems  still  to  be 
worthwhile  for  the  sake  of  completeness  to  demonstrate  this  in 
a  prototype  system. 

Also  the  modeling  of  the  pilot’s  resources  and  behavior  will  be 
further  completed  in  order  to  further  improve  timeliness  and 
goal-conformity  of  the  aiding  actions  of  the  system. 

Another  aspect  of  further  research  is  the  integrity  issue  with 
respect  to  cognitive  assistant  systems.  It  turns  out  that 
knowledge-based  systems  with  situation  analysis  capability 
bring  with  the  capability  of  self-diagnosis,  which  can  be 
exploited  for  integrity  purposes.  This  capability  provides  the 
potential  to  get  rid  of  the  necessity  of  abundant  test  runs  for 
most  part  of  the  system,  which  could  be  inevitable  to  prove  for 
sufficient  (lawlessness  in  software  and  data  bases. 

Conclusion 

Allocating  pilot  tasks  to  automatic  devices  and  improving 
aircraft  performance  and  mission  capabilities  by  inereasing  use 
of  computing  power  and  resulting  system  complexity  pose  new 
demands  on  the  cockpit  crew. 

CAMA  has  demonstrated,  that  cognitive  capabilities  within  a 
technical  system,  behaving  in  a  manner  similar  to  an  additional 
crew  member,  improve  the  effectiveness  of  the  cockpit  crew  as 
well  as  the  mission  effectiveness.  Aiding  functions  for  better 
situation  interpretation  as  well  as  planning  and  decision-making 
provide  a  safer  and  more  effective  conduction  of  the  flight.  The 
benefit  of  the  human-centered  approach,  the  cross-monitoring 
between  man  and  machine  are  shown  even  in  the  environment 
of  complex,  highly  automated  modern  flight  decks  of  transport 
aircraft  for  military  missions. 
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1.  SUMMARY 

Decision  support  has  been  identified  as  a  focus  of 
attention  for  the  mid-life  upgrade  of  the  Command  and 
Control  System  (CCS)  of  the  HALIFAX  Class  frigate, 
anticipated  in  the  2005-2010  timeframe.  To  support 
this  effort,  we  are  exploring  concepts  for  the  design  of 
computer-based  decision  aids  that  can  be  integrated  into 
the  CCS  to  assist  operators  in  their  tactical  Command 
and  Control  (C2)  activities.  To  date,  the  emphasis  has 
been  on  investigating  algorithms  for  aiding  operators 
with  their  decision  making  tasks,  including  picture 
compilation,  situation  and  threat  assessment,  and 

resource  management.  To  support  this  effort  and 
develop  an  integrated  set  of  aids,  a  systematic 

framework  is  now  needed  to  identify  the  requirements 
that  an  effective  computer-based  decision  support 
system  (DSS)  should  satisfy.  Determining  support 
interventions  in  the  work  environment  of  tactical  C2  is  a 
challenging  problem  due  to  a  number  of  its 

characteristics  (e.g.,  time-critical  decision  making  under 
uncertainty,  high  risk,  dynamism,  social  cooperation, 
large  amounts  of  data,  unanticipated  variabilities)  that 
are  similar  to  those  observed  in  other  complex 

sociotechnical  systems  in  the  civilian  world,  while 
introducing  new  ones  (e.g.,  organised,  intelligent  threats 
to  the  mission,  limited  advance  knowledge  of  threats). 
Whatever  the  chosen  analysis  and  design  framework,  it 
should  be  tailored  to  the  unique  demands  imposed  by 
these  factors.  This  paper  reviews  a  range  of  concepts 
being  explored  for  the  design  of  the  DSS,  focusing  on 
automation,  cognitive  and  methodological  issues. 
Automation  concepts  address  principles  and  paradigms 
for  decision  support.  Cognitive  concepts  deal  with  the 
cognitive  behaviours  which  the  DSS  must  exhibit  and/or 
support.  Methodological  issues  are  concerned  with  a 
specific  framework  for  determining  DSS  requirements 
known  as  Cognitive  Work  Analysis.  It  is  a  constraint- 
based  approach  to  modeling  a  complex  sociotechnical 
work  system  that  provides  an  overarching  framework 
and  a  variety  of  flexible  conceptual  tools  for 
systematically  investigating  the  space  of  design 
interventions  in  the  joint  human-machine  system.  The 
paper  also  identifies  a  high-level  framework  for  the 
immediate  design  goal,  a  DSS  that  can  support 


operators  with  data  fusion,  situation  and  threat 
assessment,  and  resource  management. 

2.  INTRODUCTION 

Tactical  decision  making  requires  a  number  of  data  and 
information  processing  tasks  to  be  continually 
performed  in  real  time  in  a  complex  dynamic 
environment.  This  calls  for  a  high  degree  of 
coordination  among  combat  system  operators  to  achieve 
the  various  objectives  in  a  timely  manner. 
Unfortunately,  there  are  a  number  of  factors  that  will 
increasingly  challenge  operators  to  perform  effectively. 
Examples  include  technological  advances  in  threats 
(e.g.,  higher  speeds,  smaller  signatures,  multimode 
guidance),  the  increasing  tempo  and  diversity  of  open- 
ocean  and  littoral  scenarios,  and  the  growing  volume  of 
data  and  information  pertinent  to  a  ship’s  area  of 
interest  that  needs  to  be  processed  under  time-critical 
conditions.  The  littoral  environment,  in  particular, 
imposes  a  number  of  difficulties  [1],  including 
significant  reductions  in  the  size  of  the  battlespace  due 
to  geographical  constraints,  increased  numbers  of 
threats,  sensor  and  guidance  system  degradation,  and 
congestion  from  commercial  air  and  surface  traffic 
which  results  in  increased  uncertainties  in  identification 
and  deconfliction. 

Such  developments  increase  scenario  complexity  and 
impose  higher  processing  demands  on  operators,  while 
reducing  the  time  available  for  decision  making. 
Operators  could  be  forced  into  the  unmanageable 
situation  of  having  to  make  extremely  rapid,  complex 
decisions,  based  on  a  limited  understanding  of  the 
tactical  situation,  for  which  they  are  accountable.  For 
example,  a  key  difficulty  for  the  ship’s  commander  is: 
failure  to  resolve  situation  uncertainty  and  hesitation  to 
act  may  lead  to  a  missile  hit;  however,  rapid  reaction  to 
what  appears  to  be  a  threat  may  lead  to  undesirable 
consequences. 

The  purpose  of  the  CCS  is  to  support  human  operators 
in  best  utilising  the  fighting  capabilities  of  the  ship.  An 
inevitable  conclusion  from  developments  in  modern 
warfare  is  that  future  shipboard  CCSs  must  provide 
better  or  new  kinds  of  support.  Unfortunately,  current 
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operational  systems  generally  provide  little  decision 
support  in  complex,  highly  dynamic  scenarios.  For 
example,  one  can  envisage  computer-based  tools  that 
automate  tracking  to  speed  up  reactions;  provide 
context  dependent  cues  to  help  focus  attention;  provide 
planning  and  threat  analysis  tools  to  assist  in  decision 
making;  present  a  common  force-level  tactical  picture; 
and  assign  weapons  under  human  veto. 

The  Data  Fusion  and  Resource  Management  Group  at 
Defence  Research  Establishment  Valcartier  (DREV) 
have  for  several  years  investigated  algorithms  to 
augment  or  enhance  existing  CCS  capabilities.  The 
emphasis  has  been  put  on  automating  various  functions 
associated  with  picture  compilation,  situation  and  threat 
assessment,  and  resource  management,  with  the 
operator  assumed  to  play  a  mostly  passive,  supervisory 
or  monitoring  role.  As  a  result,  operator-in-the-loop 
issues  and  their  impact  on  system  design  have  not 
previously  received  detailed  consideration.  DREV  is 
now  exploring  concepts  for  the  design  of  a  computer- 
based,  real-time  DSS  that  can  be  integrated  into  the 
ship’s  CCS.  Importantly,  this  extends  the  scope  of  the 
problem  to  include  human-machine  interaction  and 
team-machine  collaborative  issues,  particularly  where 
higher  level  cognitive  processing  requiring  human 
judgement  and  decision  making  is  involved. 

While  our  concern  here  is  only  with  concepts  for 
computer-based  decision  support,  there  are  a  number  of 
other  areas,  related  to  decision  support  interventions, 
e.g.,  training,  selection,  manning  levels,  organization 
and  layout,  where  improvements  can  also  be  expected  to 
positively  impact  tactical  C2  performance.  In  fact,  the 
evident  interdependencies  of  the  various  approaches 
indicates  that  they  need  to  be  jointly  analysed  to 
establish  trade-offs  and  shortfalls. 

This  paper  reviews  a  range  of  concepts  being 
investigated  for  DSS  design.  It  is  organised  as  follows. 
Section  3  describes  the  work  environment  of  shipboard 
C2.  Section  4  examines  a  wide  range  of  automation 
issues  related  to  providing  computer-based  decision 
support  in  this  work  environment.  A  cognitive  model  of 
Command  and  Control  is  presented  in  Section  5. 
Section  6  exposes  the  model-based  framework  for 
design.  Section  7  discusses  the  current  version  of  a 
high-level  framework  for  a  decision  support  system  for 
shipboard  Command  and  Control.  Finally,  Section  8 
provides  conclusions.  Various  concepts  are  only  treated 
briefly  due  to  a  lack  of  space.  A  fuller  treatment 
appears  in  [2]. 

3.  CHARACTERISTICS  OF  SHIPBOARD  C2 

Most  tactical  decision  making  in  a  modern  frigate  is 
performed  within  the  ship’s  Operations  Room.  There,  a 
team  of  combat  system  operators  (CSOs)  interact  with  a 
CCS  through  consoles  and  effect  action  by  means  of 
various  ship  combat  subsystems  (Fig.  1).  They  perceive 
and  interpret  information  available  from  ownship 


sensors  (organic  data)  or  data  linked  from  other 
cooperating  platforms  (non-organic  data),  and  plan  and 
conduct  mission  operations. 


Figure  1.  Interaction  with  ship’s  combat  system 


We  give  a  brief  description  in  this  section  of  various 
elements  of  shipboard  C2,  concentrating  primarily  on 
situation  representation.  This  is  to  expose  key  elements 
of  the  domain’s  complexity. 


3.1  Overview 

A  ship’s  command  structure  is  typically  organised 
hierarchically.  The  CSO  team  is  divided  into  sub¬ 
teams,  generally  along  warfare  areas  (e.g.,  air,  surface, 
sub-surface),  with  immediate  control  exercised  by  a 
sub-team  supervisor,  and  with  the  Commanding  Officer 
(CO)  being  responsible  in  all  respects. 

Operators  monitor  information  disseminated  to  and 
from  other  units  at  sea  and  ashore,  communicate  with 
each  other  and  provide  feedback  by  means  of 
headphones.  In  addition,  stateboards  disseminate 
current  information  on  perceived  threats  and  assist  in 
activating  pre-planned  responses  to  highly  time-stressed 
events  such  as  the  sudden  detection  of  an  anti-ship 
missile  (ASM)  flying  towards  the  ship. 

C2  tasks  require  demanding  perceptual  and  cognitive 
processing  (monitor,  detect,  assess,  diagnose,  plan,  act) 
to  be  continuously  performed.  For  example,  with 
respect  to  the  event  sequence  from  “birth”  to  “death”  of 
a  single  contact  (track),  operator  activities  span  the 
moments  from  first  contact  detection,  its  investigation 
and  evaluation  in  the  context  of  the  current  mission, 
development  of  one  or  more  courses  of  action,  to  a 
course  of  action  decision,  potentially  involving  an 
engagement  and  monitoring  of  that  engagement. 

Data  from  sensors  and  other  sources  are  continually 
processed  to  determine  a  contact’s  kinematics  (position, 
velocity,  etc.),  classify  it  at  various  levels  of  specificity 
(e.g.,  surface  combatant,  name  of  contact),  and  evaluate 
it  to  decide  whether  it  is  a  neutral,  a  friend,  or  a  threat. 
A  given  contact  may  undergo  several  changes  in 
evaluation  over  its  lifespan  -  pending,  unknown, 
assumed  friend,  friend,  neutral,  suspect,  hostile  [3]  -  as 
new  pieces  of  evidence  are  acquired  and  integrated. 

Contact  evaluation  may  require  a  variety  of 
investigative  actions  to  resolve  uncertainty  (e.g.,  send  a 
helicopter  for  closer  surveillance;  manoeuvre  the  ship 
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and  observe  the  contact’s  response;  request  additional 
information  from  a  participating  unit).  This  may 
involve  trading  off  seeking  additional  information 
against  the  time  available  for  implementing  a  specific 
action.  A  potentially  hostile  platform  must  be  evaluated 
for  its  capability  to  detect,  track,  mount  an  attack  or 
play  a  role  in  an  attack,  and  defend  itself  or  be 
defended.  Its  status  and  behaviour  must  be  monitored, 
its  assessed  intentions  understood,  and  various 
anticipatory  decisions  and/or  actions  taken  in  readiness 
for  an  attack. 

Even  if  it  is  evaluated  as  a  threat,  a  contact  may  never 
be  engaged.  For  example,  it  may  be  deploying  its 
weapons  outside  the  engagement  range  of  ownship’s 
weapons.  Even  if  it  does  come  within  the  ship’s 
engageability  envelope  there  could  still  be  a  conscious 
decision  not  to  engage,  for  example,  because  of  rules  of 
engagement,  risk  to  a  friend  or  neutral  in  the  line  of 
engagement,  or  a  desire  to  remain  covert  in  some  way. 

Beyond  these  snapshots  related  to  processing  a  single 
contact,  at  any  moment  numerous  contacts  may  have 
been  detected,  and,  in  addition  to  dividing  their 
attention  between  them,  operators  may  need  to  identify, 
monitor  and  interpret  a  variety  of  intercontact  relations. 
For  example,  functional,  spatial  or  temporal  groupings 
of  contacts  allow  structured  representations  of  the 
tactical  picture  at  different  levels  of  abstraction.  The 
operator  can  use  this  to  ease  information  processing 
load  by  zooming  in  or  out  in  the  level  of  detail  in  his  or 
her-mental  situation  picture.  Threats  can  be  prioritised 
for  processing  and  resolving  weapon  contention. 

Additional  processing  demands  arise  from  the  need  to 
continuously  fuse  data  from  a  variety  of  organic  and 
non-organic  sources  (radar,  electronic  support 
measures,  infrared  search  and  track,  identification  friend 
or  foe  transponder  responses,  data  links,  and 
intelligence).  The  resulting  information  is  used  to  build 
a  coherent  Maritime  Tactical  Picture  (MTP)  of  the 
ship’s  region  of  interest.  Difficulties  with  picture 
compilation  are  related  to  the  imperfect  nature  of  the 
data  (which  can  be  uncertain,  incomplete,  imprecise, 
inconsistent,  or  ambiguous,  or  some  combination  of 
these,  due  to  limited  sensor  coverage,  report 
ambiguities,  report  conflicts,  or  inaccuracies  in  sensor 
data  [4]),  its  quantity  and  its  timeliness.  The  amount 
and  quality  of  data  and  information  is  also  affected  by 
deliberate  interference  and  deception  by  the  enemy 
(e.g.,  by  using  decoys). 

3.2  Problems  and  Opportunities 

While  the  above  focused  exclusively  on  contact  related 
problems  in  the  tactical  picture,  there  is  in  fact  a  variety 
of  other  situation  features  that  may  alert  a  commander 
to  other  types  of  pending  “threats”  in  the  environment, 
both  internal  and  external  to  the  ship,  and  which  may 
also  need  to  be  “tracked”  and  assessed. 


Context  independent  examples  include:  the  operational 
status  of  the  ship’s  equipment  and  fighting  resources 
(e.g.,  inventory  of  missiles  and  shells,  status  of  sensors 
and  weapons);  logistics  and  personnel  status;  social  and 
political  status.  Another  example  is  related  to  plan 
execution  which  must  be  monitored  for  execution  error, 
action  outcome  uncertainty,  or  unanticipated  variability 
in  the  action  environment.  Context-dependent  examples 
include  geographical  and  environmental  constraints  of 
the  littoral  environment  that  can  significantly  reduce  the 
size  of  the  battlespace  or  degrade  weapon  and  sensor 
performance  [1]  (e.g.,  an  increase  in  the  number  of  false 
alarms  of  a  sensor  unless  its  detection  threshold  is 
lowered,  limits  in  ship  manoeuvrability  that  impact 
feasibility  of  a  particular  countermeasure). 

These  examples  indicate  that  an  operator  may  need  to 
understand  the  impact  of  a  host  of  potential  problems  on 
the  mission.  We  define  a  problem  to  be  a  feature  of  the 
situation  that  has  the  potential  to  negatively  impact  the 
achievement  of  one  or  more  goals  or  which  should  at 
least  alert  a  decision  maker  to  consider  a  change  in  the 
way  these  goals  are  being,  can  be,  or  should  be 
achieved.  A  problem  therefore  represents  an  important 
goal-relevant  property  of  the  environment  in  that  it  can 
shape  some  aspect  of  an  operator’s  behaviour.  The 
detection  of  a  problem  signals  a  possible  need  for 
corrective  measures  to  avoid  or  resolve  the  problem. 

Another  important  type  of  goal-relevant  property  for  an 
operator  interacting  with  a  complex,  dynamic 
environment  is  related  to  opportunities.  An  opportunity 
is  defined  as  a  feature  of  the  situation  that  represents  a 
possibility  to  achieve  one  or  more  goals,  or  to  accelerate 
their  achievement,  or  to  resolve  the  obstacles  to  their 
achievement.  Opportunities  may  present  themselves 
fortuitously  and  unexpectedly,  or  they  may  be  planned 
for  as  part  of  purposeful  action.  Whereas  a  problem  can 
be  thought  of  as  a  behavioural  constraint,  recognition  or 
identification  of  an  opportunity  is  an  event  that  offers 
potential  for  enlarging  the  degrees  of  freedom  for  that 
behaviour.  For  example,  a  particular  geographical  or 
environmental  feature  may  offer  an  opportunity  for 
concealing  detection  from  the  enemy.  In  some  cases, 
there  may  be  a  cost  attached  to  taking  advantage  of  an 
opportunity  (e.g.,  manoeuvring  the  ship  from  a  pre¬ 
planned  course  to  take  advantage  of  terrain  masking  to 
hide  from  a  threat,  which  intelligence  sources  have 
suddenly  signaled,  uses  fuel  but  increases  the  chances  of 
ship  survivability).  This  cost  may  need  to  be  estimated 
as  a  precursor  to  a  decision. 

We  suggest  in  Section  5.2  that  the  three  dynamic 
elements,  consisting  of  goals,  problems  and 
opportunities,  along  with  their  dynamic  relations, 
provides  a  basis  for  hierarchically  structuring  a  situation 
representation  for  tactical  decision  making  that  has 
psychological  relevance  and  offers  cognitive  processing 
efficiencies  for  the  operator.  This  structuring  also 
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underlies  our  cognitively  based  C2  model  presented 
later. 

3.3  C2  as  a  Complex  Sociotechnical  System 

We  have  seen  that  tactical  C2  involves  complex, 
dynamic,  real-time,  data-  and  goal-driven  multi-tasking. 
Goals  are  continually  created,  prioritised  and  steps 
taken  toward  their  achievement.  C2  possesses  many 
features  that  usually  characterise  a  complex 
sociotechnical  system.  Such  work  systems  are  generally 
characterised  by  (e.g.,  see  [5-6]):  uncertainty; 
dynamism;  team  work;  stress;  risk;  an  open  environment 
that  imposes  variable  and  unpredictable  demands;  large 
amounts  of  data  to  process  and  high  potential  for 
sensory  overload;  imperfect  data;  human  interaction 
mediated  via  computers;  complex  multi-component 
decision  making.  Military  C2  rates  highly  highly  in 
each  of  these  dimensions.  It  also  possesses  other 
characteristics  that  seem  to  distinguish  it  from  civilian 
sociotechnical  systems  (e.g.,  nuclear  power  plants,  air 
traffic  control).  These  differences  stem  from  the 
harshness  of  the  environment  in  which  it  operates  (e.g., 
having  to  deal  with  an  intelligent  threat  with  a  goal  of 
denying  information,  or  conveying  false  information). 

It  is  evident  that  an  open  system  like  the  C2  work 
domain  must  be  highly  responsive  to  variabilities  which 
are  very  difficult,  if  not  impossible,  to  completely 
anticipate  or  predict.  Problems  or  opportunities  can  be 
detected  at  indeterminate  times,  resulting  in  the  need  for 
dynamically  shifting,  multiple  goals.  Such  work 
systems  must  inevitably  be  highly  adaptive.  Rasmussen 
et  al.  [7]  claim  that  the  modeling  an  adaptive  work 
system  depends  on  identification  of  its  behaviour¬ 
shaping  constraints.  Our  decomposition  into  goals, 
problems  and  opportunities  is  a  step  in  this  direction. 
We  shall  return  to  this  observation  in  Section  6. 

In  practice,  of  course,  complexity  can  vary,  depending 
on  the  specific  context  and  nature  of  the  conflict 
(varying  from  low  to  high  intensity  levels  in  open-ocean 
and  littoral  areas).  Also,  complexity  in  a  given  situation 
is  likely  to  be  perceived  differently  by  individual 
operators  depending  on  their  role,  the  nature  of  their 
individual  processing  tasks,  workload,  and  experience. 

Finally,  with  increasing  pressure  to  reduce  the  through- 
life  costs  of  future  naval  platforms,  coupled  with 
demographic  data  suggesting  a  reduction  of  available 
personnel,  we  note  that  various  navy  research  efforts  in 
the  US  and  UK  are  already  examining  the  problem  of 
leaner  operator  manning  of  ships  (cf.,  for  example,  [8]). 
Reduced  manning  may  have  the  effect  of  further 
increasing  complexity  as  there  will  be  fewer  people  to 
share  the  processing  load. 


4.  Automation  ISSUES 

4.1  Overview 

Natural  questions  to  ask  concerning  computer-based 
aids  for  improving  operational  effectiveness  of  decision 
making  in  the  Operations  Room  are:  which  operator 
activities  and  positions  need  to  be  aided,  why,  when, 
and  how?  Answers  should  be  based  on  an  appropriate 
system  development  philosophy. 

It  appears  from  our  discussion  in  Section  3  that 
computer-based  support  should  be  highly  beneficial,  if 
not  a  must,  for  most,  if  not  all,  CSO  positions.  For 
example,  human  information  processing  is  subject  to  a 
number  of  limitations  and  deficiencies,  such  as  finite 
cycle  time,  limited  working  memory,  limited  ability  to 
perceive  and  process  information  and  cognitive  biases 
[9].  It  is  also  negatively  impacted  by  environmental 
factors  or  stressors  and  almost  random  mistakes  (errors 
of  judgement)  or  slips  (errors  of  execution)  [10]. 
However,  there  is  ample  evidence  in  the  literature  that 
designing  truly  supportive  technology  is  a  very 
challenging  problem,  particularly  at  the  level  of  aiding 
the  human’s  cognitive  processing  (cf.  [5,11-15]).  As 
examples,  the  burden  associated  with  supervising 
automation  as  it  performs  an  offloaded  task  can 
outweigh  the  benefits  to  improved  performance  [13]; 
performance  decrements  can  result  from  automation- 
induced  complacency  (over-trust)  [14];  a  partially 
automated  system  can  induce  more  errors  in  cases 
where  its  knowledge  is  incompetent  than  if  the  user  is 
left  in  the  loop  and  the  system  simply  critiques  the 
user’s  performance  [15].  It  has  been  suggested  that 
when  tools  dominate,  rather  than  constrain,  the  joint 
human-machine  system,  the  designer  runs  a  strong  risk 
of  solving  the  wrong  problem,  and  of  creating  new 
problems  and  undermining  existing  work  strategies  [5]. 

The  form  and  variety  of  computer-based  support 
evidently  needs  to  be  carefully  tailored  to  an  operator’s 
role,  depending  on  the  nature  and  mix  of  the  perceptual 
and  cognitive  processing  involved,  and  ideally  be 
capable  of  personal  adaptation  to  suit  the  variability  in 
support  requirements  of  the  position.  The  various 
processes  of  an  operator’s  tasks  need  to  be  established, 
decomposed  into  sub-processes,  and  decisions  made 
about  which  of  these  are  candidates  for  receiving  some 
kind  of  support.  An  important  consideration  is  the 
relative  capabilities  of  humans  and  machines  for 
performing  various  tasks  [16]  (e.g.,  the  human  is 
generally  considered  better  at  inductive  or  common- 
sense  reasoning  tasks,  whereas  the  machine  is  better  at 
deductive  reasoning). 

4.2  Aiding  Metaphors 

An  aiding  metaphor  is  concerned  with  how  support  is 
provided.  The  aid  acting  as  a  prosthesis  or  as  a  tool  are 
two  extremes  that  lead  to  two  very  different  metaphors. 
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The  differences  are  related  to  the  role  of  the  aid  in  the 
decision  process. 

The  prosthetic  approach  focuses  on  decision  outcomes. 
The  aid  essentially  replaces  the  operator  in  some  way  or 
compensates  for  human  deficiencies  in  reasoning  or 
problem  solving.  It  does  this  by  prescribing  the 
“correct”  decision  or  output  given  its  inputs.  Expert 
systems  that  provide  advice  or  decision  outcomes 
typically  fall  into  this  category.  The  operator  is  largely 
out  of  the  loop  and  plays  a  mostly  passive  role.  A 
frequent  criticism  of  this  approach  is  that  it  leads  to 
brittle  systems,  because  of  limitations  in  their  encoded 
domain  knowledge  and  assumptions  that  narrowly 
bound  their  model  of  real-world  complexity.  This 
makes  them  prone  to  poor  performance  in  the  face  of 
environmental  variability  that  has  not  been  anticipated 
by  the  system  designer.  There  is  a  large  literature  in  the 
cognitive  engineering  and  naturalistic  decision-making 
communities  [5,17]  arguing  for  an  alternative  approach. 

In  the  decision-aid-as-tool  metaphor,  focus  is  on  the 
decision-making  process  itself.  The  aid  is  considered  a 
tool  in  the  hands  of  a  competent  but  resource  limited 
agent  [18].  There  is  sufficient  flexibility,  however,  for 
it  to  adapt  to  a  novice,  with  limited  experience  (or 
maybe  just  a  battle-fatigued  expert!).  Importantly,  the 
operator  plays  an  active  role  and  the  tool  assists  in 
accordance  with  his/her  support  requirements.  Design 
emphasis  is  on  supporting  the  strengths  and 
complementing  the  weaknesses  of  the  operator. 
Moreover,  support  is  provided  for  the  operator’s 
naturally  preferred  strategies  (instead  of  enforcing  a 
normative  or  prescriptive  approach). 

To  better  appreciate  the  difference  between  the  two 
approaches  in  supporting  an  operator’s  situation 
assessment  processes,  consider  the  following  two  types 
of  aid:  one  that  builds  and  displays  a  situation  picture 
using  normatively  based  automated  reasoning 
processes;  and  another  that  acts  as  an  intelligent  alarm 
system,  monitoring  the  situation  and  alerting  operators 
to  the  occurrence  of  problems  in  the  mission  and 
opportunities  for  achieving  mission  goals  which  they 
have  requested  the  automated  system  to  track.  In  the 
former  aid,  automation  is  playing  a  prosthetic  role, 
effectively  replacing  the  operator.  Of  course,  it  may  be 
necessary  to  take  this  approach  for  selective  situation 
elements  in  certain  instances  simply  because  the 
operator  is  (temporarily)  inundated  by  a  large  number  of 
contacts  and  cannot  cope.  The  second  aid  acts  as  a  tool 
only.  Its  purpose  is  to  aid  the  human’s  limited 
attentional  resources.  The  design  challenge  in  this  latter 
system  is  to  ensure  that  it  does  not  generate  so  many 
false  alarms  that  it  becomes  totally  ignored  and  is 
simply  tuned  out  or  turned  off. 

Given  the  wide  range  of  expected  task  loads  in  a  ship’s 
Operations  Room  and  the  variety  of  types  of  processing 
involved  (monitoring,  detection,  assessment,  planning, 
etc.),  there  appears  to  be  a  place  and  need  for  both 


metaphors,  or  some  adaptable  hybrid  of  these  extremes, 
in  aiding  operators,  depending  on  situation  context,  the 
specific  nature  of  the  processing,  and  the  role  of  the 
operator.  For  example,  a  prosthetic  mode  would  seem 
appropriate  in  situations  where  the  operator  is 
momentarily  overwhelmed  and  Incapable  of  effective 
participation.  However,  both  designer  and  operator 
need  to  understand  how  the  aid’s  performance  degrades 
in  such  circumstances  to  avoid  the  problem  of  “the  blind 
leading  the  blind”.  Also,  the  operator’s  involvement 
must  be  determined  to  avoid,  or  at  least  limit,  the  effects 
of  “the  out-of-the-loop  performance  problem”  [19]. 
These  effects  leave  the  operator  handicapped  in  the 
ability  to  resume  control  in  case  of  automation  failure  or 
once  the  cognitive  demands  of  the  situation  have 
diminished  to  an  acceptable  human  level.  In  less 
demanding  situations,  a  decision-aid-as-tool  mode 
would  keep  the  operator  in  the  loop. 

The  “out-of-the-loop  performance  problem”  has  been 
linked  to  loss  of  situation  awareness  (SA)  and  skill 
decay  [19].  The  former  suggests  a  number  of  questions: 
When  situation  complexity  increases,  leading  to 
operator  overload,  what  aspects  of  their  environment  do 
they  continue  to  need  to  maintain  an  understanding  of? 
If  operators  are  withdrawn  from  a  decision  loop  in 
stages,  as  a  means  of  lightening  load,  what  support  for 
maintaining  SA  should  the  computer-based  system 
provide  at  each  stage  to  permit  judgements  and 
decisions  that  remain  part  of  their  role  (i.e.,  not  part  of 
automation’s)?  How  should  the  operator  be  able  to 
influence  the  behaviour  of  support  components  and  how 
much  does  the  operator  want  or  need  to  understand 
about  their  processing  (e.g.,  models  and  algorithms 
used;  assumptions  made)? 

4.3  Dynamic  Delegation  of  Authority 

Section  4.2  dealt  with  how  automated  support  is 
provided  once  the  decision  has  been  made  to  aid  a  given 
(sub-process  of  a  particular)  decision  process.  Related 
issues  include:  mechanisms  for  delegating  authority  to 
the  system  for  making  a  decision  about  the  outcome  or 
result  of  an  automated  (sub-)process;  dynamically 
triggering  a  change  in  delegation  based  on  changing 
situational  factors;  an  override  capability  when  the 
operator  and  the  system  have  overlapping 
responsibilities;  and  a  capability  to  influence  system 
behaviour  when  operators  have  delegated  or  lost 
authority. 

Two  approaches  to  task  delegation  are  adaptive 
automation  and  providing  a  fixed  variety  of  operator- 
system  modes.  Adaptive  automation  involves  a 
computer-controlled,  adaptive  allocation,  depending,  for 
example,  on  which  party  has  at  the  moment  more 
resources  or  is  the  more  appropriate  for  performing  the 
task  [20].  A  potential  problem  is  that  it  requires 
operators  to  keep  up  with  who  is  doing  what  as  the 
allocation  changes. 


Figure  2.  Operator-system  modes  of  operation 


One  possible  version  of  providing  various  modes  of 
operator-system  delegation,  which  resembles  that 
currently  implemented  in  the  HALIFAX  for  threat 
evaluation  and  weapon  assignment  (TEWA)  related 
tasks,  is  illustrated  in  Fig.  2.  Five  operator-system 
modes  of  operation  are  shown,  along  with  variations  in 
the  levels  of  work  distribution  and  synergy  between 
automation  and  the  operator  in  these  various  modes. 
The  human  selects  the  mode,  which  applies  until  mode 
transition  is  triggered  by  a  new  selection.  If  these 
modes  are  applied  at  the  system  level  (instead  of  at  the 
level  of  particular  decisions),  each  mode  implies  a  fixed 
delegation  of  authority  for  all  the  various  sub-processes 
for  which  automated  support  is  available.  The  bi¬ 
directional  support  arrows  in  Fig.  2  indicate  that  the 
support  paradigm  in  a  specific  mode  could  involve  the 
operator  in  an  active  processing  role  (decision-aid-as- 
tool  paradigm).  The  actual  support  paradigm  used  in  a 
given  mode  is  fixed,  but  it  does  not  have  to  be  the  same 
for  each  mode. 


example,  suppose  that  the  operator  decides  to  retain 
overriding  authority  for  some  types  of  decisions  but  is 
supported  in  these  decisions  by  the  system’s  processing. 
Two  possibilities  are:  the  system  processes,  decides  and 
acts  accordingly  only  if  the  operator  first  concurs;  and 
the  system  processes,  decides  and  acts  automatically 
unless  the  operator  vetoes.  The  operator  can  also 
influence  the  system’s  processing  in  sub-processes  for 
which  authority  has  been  allocated  entirely  to  the 
system  (e.g.,  by  requesting  that  a  specific  algorithm  be 
used).  There  is  maximum  synergy  between  the  system 
and  the  operator  in  cooperative  mode.  In  automatic 
mode,  the  system  has  total  authority,  but  the  operator 
can  influence  its  behaviour  and  request  information. 
Otherwise,  the  system  operates  in  complete  autonomy. 
The  independent  mode  completely  excludes  the 
operator.  It  processes  information  and  acts 
autonomously  without  consulting  the  operator. 

The  division  of  roles  between  the  system  and  the 
operator  in  the  various  modes  is  summarised  in  Table  1. 


In  the  silent/manual  mode,  the  operator  has  total 
authority.  Moreover,  the  system  is  completely  passive 
and  provides  no  support  whatsoever  to  the  operator.  In 
informative  mode,  the  system  only  provides  support, 
some  of  which  may  be  a  consequence  of  a  request  from 
the  operator;  authority  again  rests  solely  with  the 
operator.  The  operator  can  also  influence  characteristics 
of  the  support  provided.  In  cooperative  mode,  the 
system  and  the  operator  both  have  authority  to  decide 
and  act.  This  authority  may  be  divided  (e.g., 
responsibilities  for  some  judgements,  decisions  and 
actions  allocated  to  the  operator,  the  rest  to  the  system, 
depending  on  type)  or  shared  by  the  two  parties. 
However,  in  the  shared  case,  one  of  the  two  parties 
(operator  or  system)  has  ultimate  authority  to  override 
the  other.  This  requires  an  override  protocol.  For 


1.  S&jii/MuiUial 

I3ecide  and  act 

Passive 

Decide  and  act 

Influence  sy.stem  behaviour 

Suppoit 

3.  Co-ffpcraJive 

Decide  and  act 

Influence  system  behaviour 

Ovenlde  system 

Decide  and  act 

Suppoit 

Ovenide  operator 

4.  A«tOPSit»c 

Request  information 

Influence  system  behaviour 

Decide  and  act 

Provide  infomniion 

Respond  to  operator  influence 

5.  Irdlependeiit 

Passive 

Decide  and  act 

Table  1.  Roles  in  the  various  modes  of  operation 

Hybrid  approaches  encompassing  aspects  of  the  two 
described  above  can  also  be  formulated  [2]. 


2-7 


5.  A  COGNITIVE  MODEL  OF  C2 

5.1  Overview 

Command  and  Control  (C2)  is  the  process  by  which 
commanders  plan,  direct,  control  and  monitor  any 
operation  for  which  they  are  responsible.  The  cognitive 
model  of  C2  briefly  reviewed  here  should  be  applicable 
to  a  wide  range  of  settings,  involving  one  or  several 
operators  interacting  with  a  dynamic  environment.  For 
example,  we  anticipate  its  application  in  situations  from 
a  single  operator  in  front  of  a  console  to  a  team  of 
operators,  as  in  our  shipboard  application  [2].  There  is 
an  abundance  of  C2  models  in  the  literature  (cf.,  for 
example,  [21]).  Our  model  is  distinguished  by  its 
emphasis  on  psychologically  relevant  problem 
structuring  components  and  the  inclusion  of  both  data- 
and  goal-driven  behaviours,  modulated  by  a  meta-level. 
Additional  motivating  details  appear  in  [2]. 

This  type  of  model  should  play  a  role  in  applying  a 
model-based  approach  to  DSS  design  (see  Section  6). 
For  example,  it  permits  structuring  verbal  protocols 
from  operators  according  to  a  conceptual  framework 
and  provides  assumptions  about  the  nature  of  cognitive 
activities  to  guide  data  collection  in  field  studies. 

5.2  Structuring  the  Environment 


Figure  3.  Identifying  problems  and  opportunities 


We  first  present  a  framework  for  structuring  changes  in 
a  dynamic  environment  that,  on  the  grounds  of  cognitive 
plausibility  at  least,  appears  psychologically  relevant  to 
a  decision  maker  interacting  with  the  environment.  It 
represents  a  personalised  structuring,  from  some  frame 
of  reference  considered  relevant  by  the  decision  maker. 
For  example,  the  frame  of  reference  could  be  his  own,  a 
participating  unit’s,  or  his  enemy’s.  It  assumes  that  what 
the  decision  maker  wishes  to  comprehend  at  any  given 
time  can  be  phrased  in  the  language  (see  Section  3.2)  of 
the  problems  or  opportunities  the  environment  poses, 
given  the  goals  and  the  state  of  various  relations  that 
the  decision  maker  deems  relevant  among  these 
problems,  opportunities  and  goals  at  that  time.  In  short, 
they  are  specific,  time-dependent,  goal-relevant 
properties  of  the  environment  that  can  shape  the 
decision  maker’s  behaviour.  We  call  them  situation 
structures  because  they  provide  a  structured 
representation  for  understanding  the  environment  as  a 
prelude  to  action. 


The  decision  maker  could  be  interested  in  a  variety  of 
relations  among  goals,  problems  and  opportunities, 
including:  enabling  relations  (between  an  opportunity 
and  a  goal);  causal  and  subset  relations  (between  pairs 
of  problems  or  opportunities);  value  relations  (for 
prioritising  goals,  problems  and  opportunities);  and 
impediment  relations  (between  a  problem  and  a  goal). 

The  separation  of  events  into  insignificant  events, 
problems  and  opportunities,  shown  in  Fig.  3,  is  not 
really  an  event  partition.  An  event  could  represent  both 
a  problem  and  an  opportunity.  For  example,  the 
presence  of  a  particular  geographical  or  environmental 
feature  in  a  ship’s  vicinity,  which  route  planning  could 
avoid,  might  represent  a  problem  (reduced  sensor 
detection  envelope)  for  achieving  one  of  the  decision 
maker’s  goals  (optimise  detection)  but  an  opportunity 
(increased  chance  of  concealment)  for  another  (optimise 
survivability).  This  arises  from  conflicting  goals.  There 
is  also  a  potential  for  duality  between  problems  and 
opportunities.  For  example,  a  problem  in  one  frame  of 
reference  (e.g.,  his  own)  could  represent  an  opportunity 
in  another  (e.g.,  his  enemy’s).  This  structuring  in  a 
variety  of  frames  of  reference  should  be  an  important 
element  of  a  decision  maker’s  need  for  simultaneous, 
multiple  perspectives  in  understanding  the  situation  in 
some  cases  as  a  precursor  to  a  decision. 

5.3  Description  of  the  Model 

The  model  decomposes  the  C2  process  into  two  levels; 
a  lower  level  involving  the  three  processes  Perception, 
Situation  Representation  and  Action  Management,  and 
a  higher  level  consisting  of  various  Command  Meta- 
Processes  (Figs.  4-6). 


Figure  4.  Cognitive  C2  model 


The  Command  Meta-Processes  in  Fig.  4  dynamically 
manage  the  goals  and  choice  of  situation  structures,  and 
control  various  parameters  (like  frequency  with  which 
to  monitor  for  changes  in  a  specific  environmental 
feature)  and  the  sequencing  and  cognitive  control  level 
(in  the  sense  of  Rasmussen’s  skills,  rules,  knowledge 
(SRK)  taxonomy  [7])  of  lower  level  processes. 
Suspension  and  subrogation  control  strategies  serve  to 
shift  the  focus  of  attention  between  various  processes. 
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For  example,  a  process  might  be  suspended  during  its 
execution  because  a  higher  priority  process  suddenly 
demands  the  decision  maker’s  attention  or  simply 
because  it  cannot  be  performed  to  completion  at  the 
moment. 

Feedback  from  the  lower  level  can  cause  new  goals  and 
situation  structures  to  be  generated  and  old  goals  and 
structures  to  be  removed  from  consideration,  as  well  as 
new  control  strategies  to  be  employed.  Commands 
originating  from  higher-level  C2  echelons  can  also 
induce  changes  in  these  processes. 

Situation  Representation  and  Action  Management  are 
uncertainty  reduction  processes,  but  in  different  senses. 
The  first  (Fig.  5)  reduces  uncertainty  in  understanding 
the  situation.  In  this  case,  it  involves  judgements  about 
where  there  is  uncertainty,  incompleteness,  etc.,  in 
data/information  (short-term  knowledge)  and  in  the 
resolution  of  such  problems.  Where  it  is  deemed 
worthwhile  to  reduce  this  uncertainty  by  obtaining  more 
data/information,  it  can  issue  an  information  collection 
request  to  Action  Management  (e.g.,  send  a  helicopter 
for  closer  surveillance).  Action  Management  reduces 
uncertainty  in  the  selection  of  actions. 


Figure  5.  Situation  Representation 


In  either  case,  it  is  possible  to  add  the  need  to  reduce 
uncertainty  due  to  incomplete  long-term  knowledge 
(purposeful  learning  behaviour),  by  first  evaluating 
benefits  and  costs  of  seeking  this  knowledge  and  the 
likelihood  of  being  successful  in  doing  so.  Situation 
Representation  would  issue  a  request  to  Action 
Management  which  would  handle  its  own  needs  and 
those  of  Situation  Representation  with  subsequent 
feedback  to  Situation  Representation.  However,  these 
various  considerations  are  not  fleshed  out  in  the  figures. 

Situation  Representation  generates  descriptions  of  a 
dynamic  environment.  The  particular  descriptions  are  in 
terms  of  situation  structures  (defined  previously)  which 
the  decision  maker  (via  the  Command  Meta-Processes) 
dynamically  determines  to  be  relevant  for  determining 
and  managing  action.  Situation  Representation 
generates  these  structures  either  at  the  request  of 
Command  Meta-Processes  or  at  the  request  of  Action 
Management  when  it  needs  to  better  understand  specific 
situation  elements  for  planning  or  managing  the 
coordination  and  execution  of  its  actions.  Situation 


Representation  is  analogous  to  the  human  situation 
assessment  process  described  in  the  naturalistic 
decision-making  literature  [17].  It  combines  the 
situation  assessment  and  threat  assessment  processes  of 
the  technologically  centred  JDL  data  fusion  model  [22], 
but  from  the  perspective  of  the  human.  An  important 
nuance,  which  distinguishes  our  approach  from  previous 
efforts,  has  to  do  with  the  way  the  model  handles 
situation  projection  (i.e.,  extrapolation  of  the  current 
tactical  situation  into  the  future).  While  a  given  situation 
element  (goal,  problem,  opportunity,  relation)  may  well 
involve  an  aspeet  of  the  future  (e.g.,  the  problem  related 
to  a  contact  might  be  “Time  to  ship  intercept  is  less  than 
30  seconds”),  we  use  the  term  projection  in  an  action- 
oriented  sense  in  that  its  need  is  determined  within 
Action  Management  to  support  knowledge-based 
planning.  Process  sequencing  in  Situation 
Representation  essentially  follows  the  identification 
strategy  suggested  by  Fig.  3,  with  feedback  loops 
arising  from  the  need  to  resolve  problems  of  incomplete 
data,  information  or  knowledge. 


Figure  6.  Action  Management 


Action  Management  handles  all  processes  related  to 
determining  feasible  courses  of  action,  action  selection, 
and  management  of  action  execution.  Action  commands 
are  commands  to  physical  actuators  (sensors,  weapons, 
navigation,  etc.)  or  lower  levels  in  the  organisational 
hierarchy.  We  also  include  as  part  of  Action 
Management  decisions  related  to  sharing  information  or 
requesting  information  from  other  parties  in  the 
environment.  This  leads  to  the  communication  shown  in 
Fig.  4. 

The  presence  of  both  rule-based  and  knowledge-based 
processing  (in  the  sense  of  the  SRK  taxonomy  [7])  in 
Action  Management  should  be  noted.  This  is  also  the 
case  for  Situation  Representation.  For  example, 
diagnosis  could  be  entirely  rule-driven  or  employ,  in 
addition,  various  knowledge-based  heuristics  to  make 
judgements  that  reduce  uncertainty  in  situation 
understanding.  Evidence  of  both  types  of  behaviour 
have  been  found  in  the  naturalistic  decision-making 
literature.  For  example,  [23]  reports  findings  of  a  study 
in  which  they  interviewed  officers  in  the  Combat 
Information  Centre  of  an  AEGIS  cruiser  and  found  that 
the  primary  situation  diagnosis  strategy  was  essentially 
rule-based  feature  matching.  However,  in  situations  of 
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insufficient  information  or  when  the  situation  was  novel 
and  unfamiliar,  the  officers  used  a  knowledge-based 
strategy  of  story  generation. 

Finally,  we  draw  attention  to  a  couple  of  additional 
omissions  in  Fig.  4.  First,  the  complex  process  of  human 
perception  has  not  been  elaborated.  Second,  a  direct 
processing  path  between  perception  and  action  that 
would  correspond  to  Rasmussen’s  notion  of  skill-based 
behaviour  [7]  of  an  operator  is  not  shown. 

6.  TOWARDS  A  FRAMEWORK  FOR  DESIGN 

This  section  describes  key  ideas  of  a  framework  for 
designing  a  DSS  to  support  operators  in  the  tactical  C2 
process.  The  approach  uses  concepts  from  a  model- 
based  framework  pioneered  by  Rasmussen,  known  as 
Cognitive  Work  Analysis  (CWA)  [6-7].  Ongoing 
DREV  work  aims  to  establish  the  degree  of  fit  of  this 
approach  generally  to  the  C2  work  environment  and  to 
the  specific  design  goal  in  Section  7. 

6.1  Rationale:  Designing  for  Adaptation 

We  have  already  noted  that  tactical  C2  can  be  viewed  as 
a  complex  sociotechnical  system.  As  in  any  open 
system,  command  personnel  and  their  staff  have  to  deal 
with  a  large  variety  of  situations  or  events  both  internal 
and  external  to  the  ship,  from  familiar  ones  that  they 
encounter  routinely,  to  unfamiliar,  but  anticipated  ones, 
to  both  unfamiliar  and  unanticipated  ones.  The  designer 
seeking  to  support  CSOs  must  therefore  be  able  to 
alleviate  the  cognitive  demands  posed  by  each  of  these 
event  types. 

Unfortunately,  the  unfamiliar  and  unanticipated  event 
poses  a  particularly  difficult  problem.  How  does  one 
design  for  the  unanticipated?  Automated  solutions  to 
replace  the  human  in  such  situations  do  not  appear  to  be 
on  the  near  horizon.  Moreover,  in  this  approach, 
humans  are  inevitably  left  to  supervise  automation 
which  is  prone  to  failure  in  the  face  of  variabilities 
unaccounted  for  by  the  designer.  This  requires  the 
human  to  monitor  the  situation  for  problems  and 
develop  solutions  on  the  spur  of  the  moment,  usually 
without  support  for  this  automation-induced  role.  A 
simple  example  of  this  in  the  shipboard  context  is  an 
automated  TEWA  that  automatically  engages  threats 
under  certain  (quick  reaction)  conditions.  It  is 
impossible  for  the  designer  to  delineate  a  complete  set 
of  automatic  engagement  contingencies.  The  operator 
must  therefore  remain  in  the  loop  to  monitor  automatic 
engagements  for  problems  (e.g.,  a  friend  in  the  line  of 
engagement)  and  apply  immediate  corrective  measures 
(e.g.,  exercise  a  firing  veto). 

There  is  increasing  evidence  that  poorly  engineered 
automated  solutions  can  lead  to  substantial  performance 
decrements  of  the  joint  system  and  potentially 
catatastrophic  results.  We  suggest  that  a  more 
achievable  and  realistic  solution  can,  however,  be  found 


in  the  design  philosophy  underlying  CWA:  create 
computer-based  tools  that  help  workers  adapt  to 
unexpected  and  changing  demands  in  their  environment 
[6].  In  fact,  CWA  argues  that  the  primary  value  of 
people  in  the  human-machine  system  is  to  play  an 
adaptive  role  in  their  work  space,  with  the  necessary 
freedom  to  flexibly  change  their  behavioural  patterns, 
both  as  individuals  and  as  members  of  a  team,  as,  and 
when,  the  situation  demands.  Such  adaptation  can 
occur  at  several  levels,  including  the  levels  of  tasks, 
strategies  and  social  organization.  Rasmussen  [7] 
argues  that  to  do  this  designers  need  an  intimate 
understanding  of  the  behaviour-shaping  features  of  the 
domain  that  remain  invariant  even  with  its  unanticipated 
variabilities.  This  approach  is  consistent  with  the 
“decision-aid-as-tool”  metaphor  (see  Section  4.2),  but 
extends  it  by  emphasising  adaptation  to  handle  the 
demands  of  a  complex  sociotechnical  system. 

6.2  Behaviour-Shaping  Constraints 

CWA  provides  a  framework  for  supporting  a  variety  of 
levels  of  adaptation  by  adopting  a  model-based 
approaeh  to  work  analysis  and  DSS  design.  The 
framework  is  based  on  identifying  behaviour-shaping  or 
intrinsic  work  constraints  [6-7].  These  constraints 
delimit  the  set  of  productive  behaviours  (human  or 
machine)  without  overconstraining  such  behaviour, 
unless  required  (e.g.,  by  doctrine),  to  specific 
trajectories.  As  Vicente  [6]  puts  it,  “constraints  specify 
what  should  not  be  done,  rather  than  what  should  be 
done”. 

DSS  design  for  a  sociotechnical  system  is 
fundamentally  an  ill-defined  problem,  likened  to  solving 
a  jigsaw  puzzle  consisting  of  uncertain  pieces  and  an 
uncertain  goal  picture.  It  would  appear  that  the  only 
approach  is  to  engage  in  many  iterative,  bottom-up 
design  probes,  with  continual  technology  assessment 
and  user  evaluation  and  feedback  at  each  step  to  direct 
the  search  from  one  prototype  to  the  next.  However,  its 
consistent  application  is  problematic.  It  is  potentially 
very  ad  hoc,  expensive  in  both  time  and  cost,  and  can 
result  in  much  wasted  effort;  e.g.,  [25]  cites  one 
example  where  features  in  a  medical  DSS  that  were 
most  strongly  rejected  at  field  tests  were  those  included 
in  the  prototype  at  the  specific  insistence  of  doctors 
involved  in  development.  While  a  search  of  the  design 
space  that  actively  involves  user  participation  is  both 
advisable  and  essential,  the  problem  is  primarily  that 
this  process  alone  does  not  incorporate  any  mechanism 
but  feedback  and  developer  intuition  to  guide  it  [18]. 
CWA  provides  an  approach  to  overcoming  the  task- 
artifact  cycle  associated  with  pure  prototyping  [6]. 

CWA  is  a  top-down  approach  that  derives  power  from 
the  use  of  a  variety  of  models  that  effectively  help 
explore  the  design  space  efficiently.  By  focusing  on 
explicitly  representing  behaviour-shaping  constraints  in 
the  system,  these  models  have  both  descriptive  and 
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predictive  capabilities.  Their  descriptive  abilities 
permit  understanding  current  behaviour  in  the  C2 
system.  In  a  qualitative  sense,  their  predictive  abilities 
allow  the  designer  to  identify  and  anticipate  the 
consequences  of  design  interventions.  Naturally,  it  will 
be  difficult  in  applying  CWA  to  make  the  claim  that  all 
behaviour-shaping  constraints  have  been  modeled  or 
modeled  with  sufficient  fidelity.  However,  modeling 
must  be  seen  as  a  process  of  iterative  refinement.  The 
key  point  is  that  explicitly  representing  models  of  work 
constraints  provides  a  structured  way  of  recording  what 
has  been  included  [6]  and  making  design  improvements 
in  a  consistent  (i.e.,  not  ad  hoc)  manner.  Vicente  [6] 
contrasts  this  approach  with  other  normative  or 
descriptive  approaches  to  work/task  analysis. 

6.3  CWA  Framework 

We  sketch  here  the  CWA  framework.  Full  details  can 
be  found  in  [6-7].  CWA  identifies  five  different  layers 
of  behaviour-shaping  constraints  in  a  sociotechnical 
system,  each  layer  related  to  a  different  aspect  of  the 
work  done  in  the  system.  They  are  illustrated  in  Fig.  7. 

In  the  work  domain  layer,  work  space  constraints  are 
identified,  in  a  device-,  event-,  goal-,  task-  and  actor- 
independent  manner.  These  constraints  generally  define 
the  work  space’s  content  and  structure  or  its  field  of 
action.  The  control  tasks  layer  is  a  product  or  input- 
output  representation  of  the  work  goals  to  be 
accomplished  in  the  work  space  in  an  actor-independent 
manner.  Special  attention  is  paid  to  identifying 
processing  short  cuts  as  a  means  of  capturing 
variabilities  in  processing  sequences  among  actors.  The 
strategies  layer  represents  information  processing  in  the 
system.  It  models  the  category  of  generative 

mechanisms  or  processing  procedures  that  can  be 
employed  by  domain  actors,  as  well  as  the  constraints 
that  govern  strategy  adoption  and  switching,  depending 
on  context,  subjective  task  formulation  and  performance 
criteria.  The  social  organization  and  cooperation  layer 
models  organizational  constraints  and  content  and  form 
constraints  (workload  sharing,  social  organization,  etc.) 
underlying  human-human  and  human-machine 
cooperation  and  social  communication.  Finally,  the 
competencies  layer  models  the  constraints  associated 
with  the  operators  themselves,  including  their  generic 
capabilities  and  performance  limitations  and  more 
specialised  competence  requirements  of  their  various 
information  processing  activities.  It  is  worth  noting  that 
the  various  constraint  levels  have  far-reaching  work 
space  design  implications.  While  several  directly 
impact  computer-based  DSS  design,  some  are  also 
related  to  other  design  interventions  (e.g.,  training, 
selection,  sensors)  (see  [6,26]). 

As  Fig.  7  indicates,  the  degrees  of  freedom  in  the  design 
space  (i.e.,  the  space  of  feasible  design  interventions) 
are  reduced  as  each  new  layer  of  constraint  is  added. 
The  ordering  of  layers,  from  work  domain  to 


competencies,  is  related  to  an  ecological  orientation 
underlying  CWA.  The  ecological  approach  argues  that 
in  work  domains  that  impose  dynamic,  external 
constraints  on  the  goal-directed  behaviours  of  its 
workers,  it  is  necessary  that  work  domain  constraints 
(ecological  compatibility)  be  dealt  with  before  cognitive 
constraints  (cognitive  compatibility)  [6]. 
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Figure  7.  Constraint  layers  in  CWA 


Rasmussen  has  also  pioneered  a  set  of  generic, 
conceptual  modeling  tools  for  representing  constraints 
in  each  layer  [6-7].  These  are  shown  in  Fig.  7,  where 
ADH  is  the  abstraction-decomposition  hierarchy,  DL  is 
the  decision  ladder,  IFM  are  information  flow  maps, 
and  SRK  is  the  skills,  rules,  knowledge  taxonomy. 

6.4  An  Abstraction-Decomposition  Hierarchy 

The  first  phase  of  DREV’s  CWA  work  is  focusing 
primarily  on  the  work  domain  layer.  This  involves 
developing  an  ADH  for  tactical  C2.  We  provide  a  brief 
sketch  here  of  some  highlights  of  the  work.  More  detail 
appears  in  [27],  along  with  some  examples  illustrating 
the  anticipated  use  of  the  ADH  for  representing  the 
mental  model  of  an  operator  engaged  in  knowledge- 
based  processing  for  situation  representation  and  action 
management.  This  establishes  the  link  with  the  cognitive 
model  of  C2  presented  in  Section  5. 

It  is  worth  noting  that  information  hierarchies  have 
already  proved  useful  in  previous  developments  in  the 
maritime  environment.  For  example,  with  the  goal  of 
establishing  maritime  tactical  display  coding  standards, 
NATO  STANAG  4420  [3]  developed  a  tactical 
information  hierarchy  (taxonomy),  organized  in  a  tree, 
as  a  means  of  defining  “the  full  range  of  tactical 
information  required  by  the  operational  user  at  the 
command  level”.  A  variety  of  relationships  link  tree 
nodes  in  the  taxonomy,  including  classification, 
attribute  values,  and  generalisation/specialisation. 

However,  the  ADH  that  we  are  constructing  in  this  work 
belongs  to  a  quite  different  class  of  hierarchy.  Its 
unique  defining  characteristic  is  that  it  represents 
functional  relationships  between  work  domain  elements 
and  their  purposes,  as  well  as  part-whole  relationships, 
in  a  hierarchy  (Fig.  8).  Rasmussen  [7]  makes  the  case 
that  this  type  of  hierarchy  can  be  used  in  modeling  both 
causal  and  intentional  systems.  Certainly,  the  C2  work 
domain  comprises  both  causal  elements  (e.g.,  weapon 
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and  sensor  systems)  and  intentional  elements  (e.g., 
threats).  Important  benefits  of  this  purpose-,  function-, 
and  decomposition-oriented  description  stem  from  the 
fact  that:  it  exposes  the  work  space  information 
necessary  for  dealing  with  unanticipated  events  by 
presenting  structural  constraints  for  “proper”  operation 
of  domain  elements  (via  structural  means-ends  links); 
and  it  provides  a  psychologically  relevant  representation 
for  problem  solving,  varying  in  the  level  of  abstraction 
at  which  the  work  space  is  viewed  and  the  resolution  or 
level  of  decomposition  at  which  system  components  are 
under  attention  (via  part- whole  links)  [6],  Figure  8  uses 
the  five  levels  of  abstraction  Rasmussen  has  identified 
to  describe  process  control  systems:  functional  purpose, 
abstract  function,  generalised  function,  physical 
function,  and  physical  form  [6-7]. 

What  might  an  ADH  for  tactical  C2  look  like?  While 
the  work  needed  to  provide  a  substantive  answer  to  this 
question  is  still  in  progress,  some  general  characteristics 
of  the  ADH  and  the  process  for  deriving  it  are 
nonetheless  evident. 

First,  it  is  apparent  from  our  previous  discussion  of  the 
tactical  C2  work  domain  (problems,  opportunities,  etc.) 
that  the  work  space  can  benefit  from  being  partitioned 
into  a  variety  of  entity  types,  differing  in  purpose  or 
intention  (e.g.,  domain  of  threats,  domain  of 
countermeasures).  Developing  an  ADH  for  each 
separate  purpose  and  combining  these  ADHs  should 
simplify  the  analysis  (cf.  [6]).  Second,  in  view  of  the 
need  for  multiple  perspectives  in  understanding  a 
tactical  situation  (see  Section  5.2),  it  should  be  useful  to 
develop  a  separate  ADH  for  each  likely  perspective. 
Inevitably,  an  ADH  for  the  threat  from  an  own  force 
perspective  will  be  less  complete  than  that  for  own 
forces  due  to  knowledge  limitations. 

For  illustrative  purposes  only  and  to  help  the  reader 
appreciate  the  nature  of  the  ADH,  Fig.  9  provides  a 
simple  abstraction  hierarchy  only  for  contacts  in  the 
tactical  environment.  It  is  based  on  Rasmussen’s  five 
levels  of  abstraction  for  process  control. 


Decomposition  Level 


Whole  - ► 

Part 

Functional 

Purpose 

Abstract 

Function 

Why 

X 

G  eneralised 
Function 

V 

What 

X 

Physical 

Function 

V 

How 

Physical 

Form 
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To  conclude  this  section,  we  mention  that  an  iterative 
process  is  being  followed  in  developing  the  ADH. 
Preliminary  knowledge  acquisition  sessions  will  use  a 


variety  of  operational  level  documents  dealing  with  the 
principles  and  procedures  of  maritime  warfare.  This  will 
be  followed  by  interviews  of  subject  matter  experts 
(SMEs)  in  doctrine,  training,  and  engineering,  and  by 
observations  in  a  training  simulator.  This  will  be 
further  augmented  by  a  field  trip  on  a  HALIFAX  Class 
ship.  Initial  CWA  models  constructed  in  this  manner 
will  be  evaluated  by  SMEs  to  correct  misconceptions 
and  fill  in  gaps  in  the  models.  Finally,  data  will  be 
collected  in  a  training  simulator  to  test  the  validity  and 
completeness  of  the  CWA  models.  Although  this  work 
deals  only  with  the  first  CWA  layer,  a  plan  for 
conducting  a  full  CWA  is  also  being  developed. 
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7.  DESIGN  GOAL 

At  present,  in  the  HALIFAX,  computer-supported 
situation  representation  is  limited  essentially  to  threat 
evaluation  in  the  form  of  threat  ranking.  As  a  very 
simple  example,  the  capability  for  operators  to  request 
the  CCS  to  monitor  a  specific  contact  or  group  of 
contacts  for  a  certain  potentially  threatening  behaviour 
while  attention  is  shifted  to  a  more  immediately 
threatening  part  of  the  tactical  picture  does  not  exist. 
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Figure  1 0.  Operations  room  environment  with  DSS 

There  is  some  automated  support  in  the  CCS  for 
reactive  action  management  related  to  the  allocation  of 
the  fighting  resources  (weapon  allocation)  in  terminal 
engagement.  However,  there  are  a  number  of  areas 
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where  additional  support  should  be  highly  beneficial, 
particularly  in  complex  littoral  scenarios.  These  include 
support  for:  planning  at  both  the  operational  and  tactical 
levels;  doing  “what-if’  analyses  of  options  to  permit 
keeping  ahead  of  the  current  tactical  situation,  including 
visualization  of  options;  and  co-ordinating  the  use  of 
weapon  and  sensor  systems  or  evaluating  their 
effectiveness  in  the  current  environment. 

DREV’s  work  is  expected  to  lead  to  a  specification  of  a 
DSS  to  support  operators  at  least  in:  (i)  the  integration 
or  fusion  of  data  from  the  ship’s  sensors  and  other 
sources;  (ii)  the  formulation,  maintenance  and  display  of 
an  accurate  dynamic  situation  picture,  leading  to 
enhanced  situation  awareness;  (iii)  the  identification  and 
selection  of  courses  of  action  in  response  to  anticipated 
or  actual  threats  to  the  mission;  and  (iv)  action 
implementation  once  a  decision  to  act  has  been  made 
and  is  being  carried  out.  With  respect  to  particular  DSS 
capabilities,  item  (i)  relates  to  its  Multi-Source  Data 
Fusion  (MSDF)  capability.  It  will  support  perception 
activities  in  Fig.  4,  for  example  by  enhancing  the  quality 
and  coverage  of  the  processed  data  that  feeds 
perception.  Item  (ii)  relates  to  its  Situation  and  Threat 
Assessment  (STA)  capability.  It  will  support  the 
situation  representation  process  in  Fig.  5.  Finally,  items 
(iii)  and  (iv)  relate  to  its  Resource  Management  (RM) 
capability.  This  will  support  the  action  management 
process  in  Fig.  6. 

A  high-level  framework  of  the  DSS  integrated  with  the 
existing  CCS  is  shown  Fig.  10.[24]. 

8.  Conclusions 

This  paper  examined  a  wide  range  of  issues  currently 
being  investigated  for  the  design  of  a  decision  support 
system  to  assist  combat  system  operators  of  a  modern 
frigate  in  their  tactical  decision  making  and  action 
execution  activities  as  part  of  the  Command  and  Control 
process.  Automation,  cognitive  and  methodological 
issues  were  highlighted. 

Fundamental  issues  in  providing  computer-based 
decision  support  are  related  to  the  questions  of  which 
operator  roles  and  positions  need  assistance,  why,  when, 
and  how.  These  are  very  complex  questions  that  require 
a  coherent  methodology  to  be  followed  if  a  joint  system, 
comprised  of  both  operators  and  computer-based  aids, 
is  to  lead  to  improved  operational  effectiveness  in 
conducting  shipboard  Command  and  Control.  A  key 
problem  for  the  design  of  such  aids  is  that  they  must  be 
capable  of  operating  in  a  highly  dynamic  and  open 
environment  that  imposes  variable  and  unpredictable 
demands  on  operators.  Operators  must  be  able  to 
effectively  handle  the  demands  of  new  and 
unanticipated  situations  that  have  not  been  addressed  by 
the  system  designer  or  by  doctrine.  The  system  must 
certainly  support  the  operators  so  that  they  can  follow 
the  established  principles  and  recommended 
procedures.  Yet  it  must  not  overconstrain  them  so  that 


they  are  hampered  from  taking  advantage  of  their 
abilities  to  reason,  improvise,  and  respond,  while  at  the 
same  time  calling  on  the  system  for  the  support  they 
need.  This  paper  proposed  Rasmussen’s  Cognitive 
Work  Analysis  as  an  appropriate  framework  for 
achieving  these  design  objectives.  Work  is  ongoing  at 
DREV  to  investigate  its  applicability  to  the  design  of  a 
computer-based  system  to  support  operators  with  data 
fusion,  situation  and  threat  assessment,  and  resource 
management. 
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SUMMARY 

One  of  the  most  important  aspects  of  any  combat  engagement  is 
the  effective  gathering  of  intelligence  information.  To  accomplish 
this  information  gathering  in  modern  battlefields,  many  sensor 
platforms  are  deployed.  These  platforms  are  typically  stovepiped 
systems  that  operate  independently  of  each  other  and  require 
considerable  operator  intervention  to  interpret  collected 
information  and  adjust  the  collection  plan  accordingly.  With 
automated  methods,  these  sensor  platforms  could  collaborate  with 
each  other  and  share  information.  As  a  result,  the  effort  needed 
to  collect  information  would  decrease  and  the  gathered 
information  could  be  fused  together  to  increase  understanding  of 
the  overall  picture  of  the  battlefield. 

We  are  developing  a  system  that  facilitates  this  interaction  among 
platforms.  This  system  of  Measures  of  Merit  (MOMs)  implements 
metrics  across  platforms  at  various  levels  of  the  fusion  and  data 
gathering  process.  When  additional  information  is  needed,  the 
metrics  are  aggregated  to  measure  the  value  added  that  an  additional 
platform  can  make  to  the  mission  and  to  the  overall  battlefield 
perspective.  Intelligent  agents  distributed  among  the  platforms 
prioritize  the  requesting  and  gathering  of  information  from  other 
platforms  as  well  as  from  the  host  sensor  suite. 

We  are  applying  MOMs  to  the  collection,  connection,  and 
execution  chain  of  events  for  intelligence  surveillance  and 
reconnaissance  (ISR).  Our  development  focuses  on  the  interaction 
among  platforms  and  on  the  MOMs  algorithms.  In  this  paper, 
we  describe  the  MOMs  system  and  its  primary  components. 

INTRODUCTION 

In  today’s  battlefield  environment,  many  sensor  platforms  of 
varying  capabilities  are  deployed  to  gather  intelligence  information. 
The  process  of  planning,  tasking,  and  executing  these  missions  is 
currently  performed  using  sequential,  serial,  and  manpower¬ 
intensive  methods,  with  experts  in  one  discipline  vaguely  aware 
of  capabilities  of  other  disciplines.  As  a  result,  platforms  are 
optimized  singly  and  operate  independently  of  each  other. 

Current  information  gathering  operations  need  to  be  more  efficient 
and  flexible.  For  example,  when  new  information  arrives,  mission 


goals  may  change  and  require  retasking  of  assets.  Collection 
assets  may  be  incapacitated,  leaving  holes  in  information.  Since 
platforms  often  have  overlapping  coverage  areas  and  carry  sensors 
with  similar  capabilities,  platforms  may  duplicate  effort  by 
gathering  the  same  or  similar  information.  If  platforms  would 
share  their  information,  it  stands  to  reason  that  information 
gathering  would  be  optimized  across  platforms,  improving 
resource  management,  and  freeing  resources  to  gather  additional 
useful  information  they  were  otherwise  unable  to  gather. 

Figure  1  illustrates  the  concept  of  collaborative  platforms.  In  the 
figure,  two  sensor  platforms  have  a  view  of  a  Transporter  Erector 
Launcher  (TEL).  In  one  case,  platform  2  automatically  pushes 
information  about  the  TEL  to  platform  1 .  However,  the  value  of  a 
particular  piece  of  information  may  be  of  little  worth  and  the  use 
of  platform  I’s  communication  bandwidth  would  not  be  justified. 
In  another  case,  platform  1  may  ask  the  other  platform  to  look  at 
the  TEL  from  a  different  aspect  angle.  If  platform  2’s  mission 
plan  does  not  include  looking  for  TELs,  and  it  has  too  many  other 
higher  priority  tasks  to  accomplish,  platform  2  would  deny  such  a 
request.  Yet,  if  platform  1  could  in  turn  relay  information  beneficial 
to  platform  2’s  mission,  the  second  platform  may  negotiate  an 
exchange  and  accept  retasking  to  honor  the  request. 

While  information  sharing  among  platforms  is  beneficial,  current 
platforms  have  some  limitations  that  require  an  enhanced 
methodology  for  them  to  handle  this  sharing.  For  example, 
platforms  have  their  own  priorities  that  may  take  precedence  over 
gathering  data  for  another  platform.  Therefore,  rather  than 
automatically  retask  platforms,  it  is  desirable  to  allow  them  to 
consider  requests  and  refuse  them.  Before  requesting  information 
from  off-board  sources,  platforms  should  consider  the  capabilities, 
such  as  sensor  and  communications  capabilities,  of  these  sources. 
Platforms  also  require  enhancements  to  store,  access,  and  reason 
about  the  capabilities  of  other  platforms,  as  well  as  enhancements 
to  promote  robust  communications. 

In  this  paper,  we  overview  the  design  of  a  system  for  interactive 
Measures  of  Merit  (MOMs),  whose  purpose  is  to  provide  an 
automated  technique  to  share  information  among  platforms.  This 
system  consists  of  MOMs  that  evaluate  results  of  each  of  the 
functional  components  in  the  ISR  collection  process  and 
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distributed  intelligent  agents  (DIAs)  that  facilitate  interactivity 
among  the  platforms.  This  is  followed  by  a  discussion  of  specific 
MOMs  for  each  of  the  functional  components.  Finally,  we 
overview  the  benefits  of  this  system  implementation. 

OVERALL  SYSTEM  DESIGN 

The  MOMs  system  is  designed  to  make  decisions  based  on  the 
value  of  off-board  information  and  to  broker  with  other  platforms 
to  obtain  information  from  them.  The  concept  is  to  provide 
metrics  at  different  levels  of  the  data  gathering  and  fusion  process 
for  each  platform.  These  measures  can  then  be  used  to  make 
decisions  for  requesting  data,  sharing  data,  and  honoring  requests. 

We  divide  the  gathering  and  fusion  process  into  three  main 
functional  components:  collection  management  (CLM), 
connection  management  (CNM),  and  execution  management 
(EXM).  Each  of  these  functional  components  resides  on  each 
platform.  CLM  manages  resources  at  all  levels,  to  schedule 
platforms  and  sensors  in  accordance  with  mission  priorities.  CLM 
MOMs  measure  the  quality  of  the  collection  plan,  i.e.,  the 
allocation  and  scheduling  of  tasks  among  multiple  ISR  assets. 
Contributions  to  CLM  MOMs  include  the  probability  of 
successful  collection  and  the  timeliness  of  collected  information. 
CNM  manages  communications  assets,  based  on  the  demands  of 
EXM  and  CLM,  and  encompasses  communication  issues  between 
platforms  as  well  as  communications  to  the  fusion  system  itself. 
CNM  MOMs  assess  connectivity  paths  among  collection 
platforms  and  to  users.  Contributions  to  CNM  MOMS  include 
bandwidth,  links  between  platforms,  communication  errors,  the 
optimal  use  of  connection  paths,  and  the  ability  to  adapt 
interconnections.  EXM  controls  the  actual  data  collection  and 
fusion  process.  EXM  MOMs  evaluate  the  effectiveness  of  a 
platform’s  collected  information  to  contribute  to  satisfying 
mission  needs  for  a  weU-defined  fused  picture  of  the  batUespace. 
Contributions  to  EXM  MOMs  calculations  include  quality  of 
target  track  localizations  and  classifications  as  well  as  ambiguity 
and  errors  in  associating  new  measurements  to  tracks. 

Figure  2  illustrates  the  top-level  design  and  process  flow  for 
MOMs  in  the  collection,  connection,  and  execution  chain  of  events 
for  a  sensor  platform.  This  design,  which  is  replicated  for  each 
platform,  shows  the  main  system  components  and  their 
interaction,  as  well  as  interfaces  to  external  platforms  and 
functions.  To  assist  operators  with  complex  analyses  across 
platforms  and  disciplines,  the  EXM,  CNM,  and  CLM  components 
contain  MOMs  that  assess  the  quality  of  the  collection  plan, 
connectivity  paths,  and  collected  information.  DIAs  facilitate 
collaboration  and  communication,  while  feedback  loops  provide 
multiple  opportunities  to  adjust  the  collection  process  based  on 
the  assessments  and  improve  results. 

In  the  design,  the  Connection  Agent  handles  the  communications 
among  the  platform  components  and  external  interfaces.  The 
Connection  Agent  receives  the  initial  task  allocation  and  passes  it 
to  CLM,  which  locally  optimizes  its  collection  plan  against  its  task 
allocation.  The  CLM  MOM  system  evaluates  the  collection  plan 
and  CLM  makes  improvements  to  the  plan.  CLM  sends  the  plan 
via  the  Connection  Agent  to  EXM,  which  controls  the  asset  sensor 
suite  to  execute  the  collection  plan.  EXM  fuses  the  collected 
information  and  the  EXM  MOM  system  evaluates  the  effectiveness 
of  the  fused  information  with  respect  to  the  asset  mission  objectives. 
Based  on  that  evaluation,  EXM  determines  information  shortfalls. 


or  gaps.  The  Execution  Agent  determines  whether  the  platform 
can  fill  the  information  gaps,  generates  new  collection  tasks,  and 
passes  them  to  CLM  via  the  Connection  Agent.  When  the  platform 
cannot  fill  the  gaps,  the  Execution  Agent  evaluates  capabilities  of 
other  platforms,  estimates  the  value  added  of  obtaining  information 
from  these  platforms,  and  directs  CNM  to  request  the  information 
from  other  assets.  The  CNM  determines  connection  paths  and  the 
CNM  MOM  evaluates  the  planned  connection  paths  and  makes 
improvements.  Then  CNM  requests  the  information  from  other 
platforms,  which  may  fulfill  or  deny  the  request. 

Similarly,  CNM  receives  information  requests  from  other 
platforms  and  relays  them  to  EXM.  The  Execution  Agent 
evaluates  the  requests  in  light  of  the  platform’s  own  priorities 
and  determines  whether  or  not  to  honor  the  requests.  If  the 
Execution  Agent  determines  it  is  better  to  deny  the  request,  it 
then  considers  opening  negotiations  with  the  requesting  platform 
to  fulfill  the  request  in  exchange  for  needed  information  the  other 
platform  could  supply. 

When  evaluating  the  value  added  of  requesting  another  platform 
to  gather  information,  the  Execution  Agent  considers  the  following 
capabilities  of  that  platform: 

1.  Automatic  retasking.  The  effectiveness  of  retasking  assets 
based  on  changes  in  observed  events  or  a  reassessment  of 
collection  priorities. 

2.  Cooperative  multiple  asset  management.  The  effectiveness 
of  cooperating  to  optimize  multiple  sensor  resource 
management,  data  sharing,  and  distributed  fusion. 

3.  Multiple  asset  synchronization.  The  effectiveness  of 
coordinating  coverage  of  the  battlefield  while  feasibly 
meeting  collection  and  priority  requirements. 

4.  Platform  resource  management.  The  effectiveness  to 
optimize  multiple  sensor  resource  management  and 
centralized  fusion  within  a  single  platform. 

SYSTEM  INTERACTIVITY  VIA  DISTRIBUTED 
INTELLIGENT  AGENTS 

DIAs  are  essential  to  achieving  collaboration  among  the  sensor 
platforms.  DIAs  are  the  mechanism  that  supports  the  interaction 
among  multiple  distributed  sensor  platforms  by  providing  an 
infrastructure  of  small,  stand-alone  processes  that  are  flexible, 
modular,  robust,  understandable,  and  cost-effective.  These 
processes  have  many  advantages  over  standard  software  processes: 

•  Social  ability  to  interact  with  information  sources,  each 
other,  and  humans 

•  Pro-activity  and  reactivity  to  initiate  action  based  on  their 
own  reasoning,  as  well  as  perceive  their  environment  and 
quickly  respond  to  changes  in  it 

•  Task  driven  and  data  driven  to  operate  in  both  a  goal- 
directed  manner  (e.g.,  working  to  achieve  a  task)  and  in 
a  data-directed  manner  (e.g.,  initiating  an  action  based 
on  processing  results) 

•  Embedded  domain  knowledge  about  methods  for 

performing  tasks  and  knowledge  about  the  context 

•  Explicit  knowledge  representations  and  procedural 
knowledge  to  explain  an  agent’s  behavior,  both  in  terms 
of  what  it  is  doing  and  why 
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These  characteristics  make  it  easy  to  upgrade  DIAs  to  incorporate 
new  knowledge  and  capabilities  that  evolve  over  time. 

In  our  design,  each  participating  platform  has  the  following  three 
DIA  functions; 

•  Pull:  requests  data  from  other  platforms  needed  to 
optimize  fusion  performance  (as  opposed  to  receiving  a 
fixed  set  of  information) 

•  Push:  sends  pertinent  data  to  other  platforms  by 
anticipating  their  data  needs  (as  opposed  to  simply 
broadcasting  data) 

•  Refusal:  considers  and  negotiates  requests  from  other 
platforms  based  on  its  own  priorities  and  cost 

Figure  3  illustrate  the  interaction  of  the  pull  and  refusal  functions 
between  two  platforms. 

In  the  design  of  Figure  2  above,  the  Execution  Agent  reasons 
about  information  gaps  and  the  capabilities  of  other  platforms, 
then  determines  which  information  to  pull,  push,  and  refuse.  The 
Connection  Agent  performs  the  communications  functions  needed 
to  pull,  push,  and  refuse  the  information.  Specifically,  the 
Connection  Agent  handles  task  messages  among  platforms,  as 
well  as  among  functional  components  of  each  platform.  These 
task  messages  include  allocated  tasks,  task  acknowledgments,  and 
requested  data.  The  Connection  Agent  also  provides  routing,  flow 
control,  and  error  control. 

EXECUTION  MEASURES  OF  MERIT 
EXM  MOMs  have  been  the  primary  focus  of  our  MOMs 
development  thus  far.  The  purpose  of  these  MOMs  is  to  reduce 
the  operator  load  by  determining  which  targets  immediately  need 
to  receive  a  measurement  update,  estimating  how  well  a  new 
measurement  would  improve  the  target  track,  and  measuring  the 
actual  level  of  improvement.  In  a  battlefield  environment,  an 
operator  would  be  quickly  overwhelmed  with  such  computation. 

The  challenge  in  creating  such  metrics  is  two-fold.  As  an  element 
of  a  real-time  system,  the  metrics  depend  solely  on  measurements 
and  calculated  results.  Truth  is  considered  an  unobtainable 
quantity.  Therefore,  if  not  developed  properly,  the  metric  may 
routinely  provide  high  quality  scores  even  though  the  results  vary 
significantly  from  truth.  The  second  challenge  in  developing  EXM 
MOMs  is  that  they  need  to  provide  reports  on  current  track  quality, 
estimated  track  quality,  and  resulting  track  quality.  These  reports 
identify  which  tracks  need  to  receive  new  information,  which 
sensor  reports  would  be  of  value,  and  the  accuracy  of  our  estimates. 

To  meet  these  challenges,  we  are  using  three  metrics  that  when 
implemented  together  determine  instantaneous  track  quality, 
comparisons  over  time,  and  a  performance  evaluation  over  time. 
Our  metrics,  entropy  [Hamming],  the  Gap  [Stewart],  and  order 
statistics,  rely  only  unmeasured  and  computed  values  and  provide 
the  information  necessary  for  the  calculation  of  the  desired  track 
quality  scores. 

The  instantaneous  track  quality  is  provided  by  entropy.  Entropy 
is  defined  as 

H(X)  =  ilog(2nd6tlXle) 
for  Guassian  density  functions,  and 


H{X)  =  -'£P{x)logP{x) 

X 

for  discrete  density  functions.  Entropy  provides  a  measure  of  the 
degree  of  uncertainty  for  each  solution.  We  use  it  to  measure 
localization  and  classification  quality  as  well  as  measurement-to- 
track  ambiguity.  For  example,  in  Figure  4,  a  platform  using  a  multi¬ 
hypothesis  tracker  is  tracking  five  targets.  The  localization  of  a  target 
is  represented  as  a  Gaussian  distribution.  The  classification  is 
described  in  a  Bayesian  taxonomy  [Pearl],  which  can  be  interpreted 
as  a  discrete  probability  density  function  as  can  the  hypothesis  scores. 
From  this  information  we  can  determine  which  tracks  are  well 
localized  and  classified,  and  the  ambiguities  that  exist  in 
measurement-to-track  associations  and  can  be  quickly  computed. 

While  this  instantaneous  information  is  very  useful  for  providing 
elements  of  all  three  track  quality  scores,  we  are  also  interested  in 
determining  how  well  we  are  tracking  targets  over  time.  The  ability 
to  compare  results  eases  determination  of  the  value  added  of  new 
information.  While  new  data  can  change  all  of  our  results,  its 
benefit  needs  to  be  measured.  For  instance,  an  improvement  of 
half  a  percent  in  accuracy  may  be  too  little  to  warrant  reassigning 
assets  and  processing  their  results .  The  way  we  monitor  the  benefit 
of  new  data  is  through  the  Gap  metric,  which  measures  the  distance 
between  subspaces.  This  distance  is  measured  by  the  canonical 
angles  between  vector  subspaces.  By  defining  our  entropy  results 
as  sets  of  subspaces,  we  can  determine  in  which  direction  the 
greatest  amount  of  change  has  occurred.  From  this  measure,  we 
determine  the  quality  of  the  data  used  as  well  as  which  parameters 
are  most  affected  by  a  particular  sensor’s  measurements. 

One  of  the  key  elements  of  EXM  MOMs  is  the  ability  to  predict 
how  well  a  measurement  can  improve  a  target  track.  Aplatform’s 
DIA  estimates  another  platform’s  sensor  suite  and  the  tracking 
system  processes  the  information.  By  interpreting  the  change  in 
entropy,  we  determine  whether  or  not  the  measurement  is  useful. 
However,  we  need  to  compare  our  estimated  improvement  with 
the  actual  improvement.  These  errors  over  time  can  provide  a 
useful  bound  on  the  accuracy  of  the  DIA  and  its  estimation  system. 
We  are  using  order  statistics  to  create  these  bounds.  Order 
statistics  are  a  distribution  free  method  of  ranges  of  values,  in 
this  case  error  of  prediction  to  actual  results,  with  a  given 
confidence.  If  the  bounds  are  tight,  the  predictions  are  accurate. 
Larger  bounds  indicate  that  prediction  results  are  suspect.  This 
information  is  then  used  by  the  DIAs  to  determine  reliability  of 
given  platforms  and  estimation  systems. 

EXM  MOMs  provide  information  on  how  well  the  current  data 
fusion  system  is  performing.  They  also  can  indicate  the  potential 
usefulness  of  new  information.  Our  basic  implementation 
procedure  is  to  measure  the  amount  of  uncertainty  in  the  targets’ 
localizations,  classification,  and  measurement  association 
ambiguity  using  the  entropy  measure.  We  then  use  the  Gap  metric 
to  measure  behavior  of  the  data  fusion  system  over  discrete  time 
intervals.  By  using  measurement  estimates,  we  can  predict 
behavior.  Finally,  we  use  order  statistics  to  measure  the  accuracy 
of  our  predictive  capability.  This  automation  provides  a  wealth 
of  additional  information  and  conclusions  that  operators  are 
unable  to  provide  given  their  current  workload. 

COLLECTION  MEASURES  OF  MERIT 

The  purpose  of  CLM  MOMs  is  to  measure  the  quality  of  the 

collection  plan,  which  typically  consists  of  resources  that  have 
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tasks  allocated  to  them  and  a  schedule  for  achieving  these  tasks. 
In  contrast  to  EXM  MOMs,  which  are  concerned  with  the  quality 
of  the  actual  collected  information,  CLM  MOMs  are  concerned 
with  theplan  to  collect  information.  By  evaluating  the  collection 
plan,  we  can  optimize  the  use  of  collection  assets,  reduce  duplicate 
coverage,  and  close  coverage  gaps  by  tasking  alternative  assets 
to  collect  information.  By  starting  with  a  higher  quality  plan, 
collection  will  be  more  efficient  and  timely  and  reduce  the  extent 
of  revisions  that  operators  need  to  make. 

Our  approach  for  developing  CLM  MOMs  differs  greatly  from 
our  approach  for  EXM  MOMs.  For  CLM,  we  define  a  top-level 
MOM,  decompose  it  into  lower  level  MOMs,  and  decompose 
those  into  measures  of  performance  (MOPs),  measures  of 
effectiveness  (MOEs),  and  measures  of  robustness  (MORs). 
When  evaluating  the  collection  plan,  we  evaluate  low  level 
contributing  factors  to  MOPs,  MOEs,  and  MORs  and  aggregate 
these  measures  to  produce  the  top-level  measure. 

Figure  5  shows  this  CLM  MOM  decomposition  and  characteristics 
that  contribute  to  the  calculations.  The  individual  metrics  are 
combined  through  a  Bayesian  network  and  accommodate  factors 
that  change,  such  as  priorities,  through  a  simple  weighted  average 
according  to 

MOMcm  =  Wepa  MOMepa  +  WMAC  MOMmAC  +  ^CRP  MOM^RP, 
where  the  weights  are  selectable  and  must  satisfy 
wepa  +  wmac  +  wdrp  =1-0 
with  each  individual  weight  in  the  interval  [0,  1]. 

CONNECTION  MEASURES  OF  MERIT 
The  purpose  of  CNM  MOMs  is  to  measure  the  quality  of 
communications  between  platforms  and  ground  stations  with 
respect  to  the  goals  of 

•  Optimal  connectivity 

•  Value  added  of  an  additional  connection 

While  CLM  MOMs  assess  the  collection  plan  and  EXM  MOMs 
assess  information  as  it  is  collected,  CNM  MOMs  are  valuable 
in  both  the  planning  stages  and  in  real-time  management  of  the 
communication  networks.  With  respect  to  planning,  CNM  MOMs 
evaluate  the  adequacy  of  the  planned  communication  networks, 
the  value  of  adding  an  additional  data  link,  the  degradation  due 
to  anticipated  communication  delays,  and  the  impact  of  losing  a 
communication  path. 

During  real-time  collection,  the  foremost  impact  to  the  user  is 
the  timeliness  of  receiving  information.  Transmission  delays, 
alternative  communication  paths,  bandwidth,  and  compression 
affect  the  timeliness  of  getting  information  to  the  user.  In  addition, 
the  frequency  of  updating  a  scene,  target  density,  target  type  (air, 
land,  or  stationary)  and  message  formats  also  drive  the  timeliness. 
In  order  to  obtain  optimal  system  performance  during  collection, 
the  communication  processing  loads  must  be  balanced  and 
messages  must  be  dynamically  routed. 

Communication  effectiveness  determines  the  success  of 
coordinating  data  collection  between  platforms  and  provides  the 
means  for  real-time  retasking.  The  evaluation  of  communications 
connections  during  both  planning  and  execution  frees  operators 


from  performing  this  tedious  analysis.  We  can  incorporate  more 
details  into  an  automated  evaluation,  to  achieve  higher  quality 
results,  and  provide  more  timely  information  for  real-time 
communications  management. 

To  assess  connectivity  paths  among  collection  platforms,  we  are 
again  using  entropy.  This  key  information  theory  technique  can 
handle  both  continuous  and  discrete  information,  as  well  as  both 
types  of  information  combined. 

An  example  is  a  decentralized  communication  network  for  target 
tracking  that  uses  entropy  as  an  information  fusion  metric.  As  a 
scalar,  entropy  provides  a  direct  comparison  between  alternatives 
and  is  applicable  to  continuous  (e.g.,  position,  speed,  and  course), 
discrete  (e.g.,  classification  such  as  air,  land,  friendly,  hostile), 
or  joint  mixed  probability  density  functions.  The  objective  is  to 
maximize  the  information,  or  equivalently,  to  minimize  entropy. 
The  global  entropy  is  given  by 

H(global)  =  H(local)  +  H(received)  -i-H(communicated) 

where  the  local  communications  decide  which  information  to 
communicate  by  determining  the  global  impact  of  communicating 
track  information  represented  by  H(communicated). 

CONCLUSIONS  AND  RECOMMENDATIONS 

In  this  paper,  we  presented  technologies  for  a  next-generation 
system  of  interacting  sensor  platforms  that  share  information  with 
each  other.  We  introduced  the  three  main  components  in  the 
information  gathering  process:  CLM,  CNM,  and  EXM;  and 
overviewed  an  approach  for  determining  when  it  is  beneficial  for 
platforms  to  share  information. 

A  key  aspect  of  this  approach  is  the  use  of  metrics  to  measure  the 
quality  of  the  collection  plan,  gathered  information,  and 
connections.  These  metrics  are  u,sed  to  determine  the  value  added 
of  tasking  additional  assets  to  collect  more  information,  as  well 
as  to  optimize  asset  utilization.  Another  key  aspect  is  the  use  of 
DIA  technology  to  evaluate  platform  capabilities  and  their  value 
added,  negotiate  information  exchange  among  platforms,  and 
facilitate  communications  with  other  platforms.  We  also  discussed 
a  variety  of  technologies  for  implementing  the  MOMs,  including 
entropy,  the  Gap,  order  statistics,  and  weighted  sums. 

The  information  sharing  discussed  here  provides  several  benefits 
for  the  ISR  community.  First,  this  sharing  facilitates  theater- wide 
tracking,  targeting,  and  surveillance  capabilities.  The  automation 
reduces  operator  workload  for  performing  these  complex  analyses 
across  platforms  and  disciplines.  The  collaboration  among 
platforms  optimizes  asset  use,  since  assets  can  share  information 
rather  than  collect  duplicate  information.  Another  benefit  is  better 
coverage,  since  platforms  can  fill  information  gaps  and  obtain 
supplemental  information.  The  result  is  that  resources  can  be  freed 
to  obtain  other  useful  information  that  they  were  otherwise  unable 
to  obtain.  The  end  result  is  better  quality,  more  timely  information 
that  contributes  to  an  improved  overall  picture  of  the  battlefield. 

This  system  has  been  designed  and  individual  components  have 
been  tested  to  demonstrate  their  capabilities.  Our  next  step  is  to 
incorporate  EXM  MOMs  and  DIAs  into  a  multi-platform  testbed 
simulation  system  and  demonstrate  the  interactive  capabilities. 


ACKNOWLEDGMENTS 

We  thank  the  Air  Force  Army  Research  Laboratory  for  funding  this 
research  and  development  under  contract  F30602-97-C-0010.  We 
thank  the  following  for  their  excellent  contributions  to  the  MOMs 
system  concept:  Dr.  Ivan  Kadar  (Northrop-Grumman  Corp.);  Dr. 
Charles  Morefield,  Dr.  Chris  Donohue,  and  Mr.  O.  Patrick  Kreidl 
(all  Alphatech,  Inc.);  and  Larry  Nelson  (Integrated  Sensors  Inc.). 

REFERENCES 

[HammingSO]  Hamming,  R.W.,  Coding  and  Information 
Theory,  Prentice-Hall,  Englewood  Cliffs,  NJ,  1980. 

[Pearl88]  Pearl,  J.,  Probabilistic  Reasoning  in  Intelligent 
Systems:  Networks  of  Plausible  Inference,  Morgan  Kaufinann 
Publishers,  San  Mateo,  CA,  1988. 

[Stewart90]  Stewart,  G.W.  and  J.  Sun,  Matrix  Perturbation 
Theory,  Academic  Press,  Boston,  1990. 


5-6 


Figure  1.  Sensor  platforms  share  information  to  more  efficiently  cover  the  area  of  interest 


Other  Platforms 


Fused  Information 


Figure  2.  Top-level  design  for  Measures  of  Merit  (MOMs),  showing  main  components  and  their  interaction  to  assess 
collection  quality  and  adjust  the  collection  plan  to  improve  results 
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Figure  3.  Push,  pull,  and  refusal  agents  dynamically  reason  about  the  ability  of  sensor  platforms  and  interact  with 

platforms  to  share  information 


Figure  4.  EXM  MOMs  are  based  on  information  for  target  error  ellipses,  hypotheses,  and  classification 
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Figure  5.  Lower  level  measures  are  aggregated  to  form  an  overall  Collection  Management  MOM 
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1.  SUMMARY 

We  propose  in  this  article  several  multisensor  tracking 
algorithms  for  battlefield  applications.  Generally,  the 
context  is  never  taken  into  account  in  the  multisensor 
systems.  Moreover,  it  can  have  an  influence  on  the 
performance  of  the  sensors.  Contexts  can  be  for  example 
hidden  zones,  jamming,  or  meteorological  conditions. 
The  main  interest  of  the  presented  research  is  to  describe 
a  multisensor  tracking  method  which  takes  context  into 
account  and  to  use  it  for  battlefield  applications.  By 
analysing  the  context,  the  relative  confidence  in  the 
sensors  is  taken  into  account  in  the  tracking  algorithm. 
A  ponderation  coefficient  is  defined  which  represents  a 
very  simple  way  to  handle  and  a  very  flexible  tool  to 
modulate  the  relative  confidence  levels  of  sensors 
depending  on  their  own  characteristics,  their 
environment  and  those  of  the  observed  targets.  This 
method  avoids  that  the  measurements  of  no  reliable 
sensors  disturb  the  result  of  the  global  fusion.  The 
objective  is  to  have  a  robust  system  which  always 
conserves  the  target  track.  In  the  last  section  of  this 
article,  we  present  results  of  numerical  simulations 
based  on  realistic  battlefield  scenarios  in  which  a  ground 
target  moves  behind  hidden  zones.  The  proposed 
algorithm  is  compared  with  a  classical  tracking 
algorithm.  This  parallel  study  emphasizes  a  better 
performance  of  the  suggested  method. 

2.  INTRODUCTION 

The  interpretation  of  increasing  amounts  of  battlefield 
information  by  both  operator  and  systems  is  more  and 
more  difficult.  Therefore,  data  fusion,  i.e.  combining 
data  firom  several  sources  in  order  to  obtain  a  global  and 
coherent  view  on  (a  part  of)  the  battlefield  at  sensor  level 
(multisensor  data  fusion)  becomes  very  important 

In  this  paper  we  focus  on  the  multisensor  tracking 
problem.  In  multitarget  tracking  applications  [1],  [2], 
[5],  the  algorithms  proposed  for  data  fusion  are  Kalman 
filters  and  extensions  IMM  (Interacting  Multiple 
Models),  PDAF  (Probability  Data  Association  Filter), 
JPDAF  (Joint  Probability  Data  Association  Filter).  The 
assumption  that  the  characteristics  of  the  sensors  are 
known  is  always  made.  Moreover,  the  context  is  never 
taken  into  account,  that  supposes  it  is  always  in  favour 
of  the  sensor  concomitant  use.  It  is  obvious  this  fact  is 


seldom  verified.  Especially,  hidden  zones,  jamming, 
E.M.  interferences,  meteorological  conditions,  are  all 
contexts  which  influence  the  performance  of  the  sensors. 
A  multisensor  system  can  work  in  all  conditions  only  if 
the  context  is  analysed  and  if  the  fusion  process  takes  it 
into  account  in  the  algorithms.  The  main  aim  of  the 
presented  research  is  to  focus  on  the  way  in  which 
different  factors  can  modify  the  conclusion  of  classical 
filtering  algorithms. 

In  this  context,  this  paper  is  composed  of  two  important 
sections.  The  first  section  describes  two  diffoent 
methods:  a  classical  multisensor  tracking  algorithm  and 
a  more  sophisticated  multisensor  tracking  algorithm 
which  takes  context  into  account  [3],  [4].  The  second 
section  describes  the  validation  of  multisensor  tracking 
methods  for  battlefield  applications.  In  this  part,  we 
present  results  of  numerical  simulations  based  on 
realistic  scenarios  in  which  a  ground  target  moves 
behind  hidden  zones.  A  comparative  study  of  the 
proposed  algorithm  and  classical  tracking  algorithm  is 
ma^  and  demonstrates  the  benefice  of  the  suggested 
method. 

3.  MULTISENSOR  TRACKING  METHODS 

3.1  Choice  of  coordinates  system 
The  first  difficulty  in  the  implementation  of  a  tracking 
algorithm  is  to  choose  the  coordinates  system.  This 
problem  has  been  little  studied  for  the  multisensor 
systems  compared  with  monosensor  systems.  As  we 
will  see  it  later,  there  are  two  possible  approaches  for 
resolving  it.  A  first  one  consists  in  adopting  a  cartesian 
coordinates  system  for  the  representation  of  state 
variables.  Such  a  model  permits  to  represent  with 
precision  the  target  dynamic  and  requires  to  use  an 
Extended  Kalman  filter  (EKF)  because  resulting 
observation  equations  are  not  linear.  But  for  the 
degenerated  cases,  this  approach  is  limited.  The  loss  of 
one  coordinate,  like  radar  range,  can  lead  to  an  important 
global  deterioration  of  filtering  results.  The  other 
approach  consists  in  adjusting  an  independent  filter  to 
each  coordinate.  If  it’s  necessary,  polar  to  cartesian 
coordinates  transformation  can  be  realized  after  filtering. 
This  last  approach  has  been  selected  in  this  p^wr 
because  it  presents  two  main  advantages:  used  filters  in 
this  one  are  simple  (linear  Kalman  filter)  and  need  no 
inversion  matrices;  it  is  more  robust  to  perturbations. 
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Subsequently,  we  consider  a  multisensor  system 
composed  of  two  sensors.  Filter  equations  are  written  in 
considering  only  one  of  the  coordinates  (range  or 
azimuth)  because  the  method  is  exactly  the  same  for 
each  coordinate. 

3.2  Description  of  a  classical  multisensor 
tracking  method 

3.2.1  Synchronous  case 

In  this  section,  the  considered  system  is  synchronous 
and  the  described  algorithm  is  sequential  [6],  [7].  For  a 
system  composed  of  two  sensors,  state  equations  can  be 
split  up  into  two  sub-systems: 

Dynamic  Xk  =  F  xn  -f  Vk 

Sensor  1  yt  =  Hi  Xk  +  bi 

Dynamic  xi  =  Xk 

Sensor  2  yi  =  H2  xt  +  bk 

where  Xj  is  the  target  state  (position,  velocity, 
acceleration),  F  is  the  transition  matrix,  Vk  is  the  state 
noise  whose  covariance  is  defined  by  E(Vivj)  =  Q  5(i,j) 
with  Q  constant,  5  the  Kronecker  symbol.  At  time  k, 
sensor  1  measurement  is  designed  by  yi  and  sensor  2 
measurement  by  yi  .  Measurements  are  connected  to 
state  with  observation  matrices  Hi  and  H2.  Measurement 
noises  of  each  sensor  are  not  correlated  and  are  noted  bk 
and  bL  Their  variance  are  respectively  defined  by 
E(bi  bj‘)  =  Ri  5(i,j)  and  E(l^^bf ‘)  =  R2  6(i,j). 

Consequently,  this  decomposition  suggests  making 
treatment  in  three  successive  steps:  one  prediction 
process  and  two  estimation  processes. 

SlscJ. 

With  current  state  Xk-i/k-i  and  covariance  Pk-i/k-i, 
estimated  at  time  k-1,  prediction  is: 

A  /V 

Xk/k-l  =  F  Xk-l/k-l 
Pk/k-1  =  F  Pk-l/k-1  F  -t-  Q 

where  Xk/k-l  is  the  state  prediction  at  time  k  and  Pk/k-i 
the  associated  covariance. 

Stgp2 

The  updated  state  estimate  and  the  associated  covariance 
are  calculated: 

Xk/k  =  Xk/k-l  +  Ki(k)(yi  -  Hi  Xk/k-l) 

Pk/k  =  (I  -  Ki(k)  Hi)  Pk/k-i 

where  Ki(k)  is  the  Kalman  gain  for  sensor  1 
measurement  and  is  given  by  the  formula: 

Ki(k)  =  Pk/k-i  hT  (Hi  Pk/k-i  hI  -r  Ri)’‘ 


Stgp3 

In  this  third  process,  we  identify  the  predicted  state  with 
the  estimated  state  at  the  previous  step  and  the  state 
prediction  covariance  with  the  updated  state  covariance 
calculated  before.  So,  the  second  estimation  leads  to  the 
following  expressions: 

Xk/k  =  Xk/k  +  K2(k)(yi-H2$k/k) 

Pk/k  =  (I-K2(k)H2)PL/k 

where  K2(k)  is  the  Kalman  gain  for  sensor  2 
measurement  and  is  given  by  the  formula: 

K2(k)  =  Pi/k  Hi  (H2  Pl/k  Hi  R2)‘' 

In  the  synchronous  case,  it’s  also  possible  to  use  a 
specific  Kalman  filter  [8]  which  is  more  suitable  for  a 
multisensor  system.  But,  this  method  presents  an 
important  drawback:  it  requires  to  inverse  the  covariance 
matrices  Ri  and  R2  for  the  Kalman  gain’s  calculation. 
It’s  the  reason  for  which  we  have  retained  the  above 
sequential  algorithm. 

3.2.2  Asynchronous  case 

In  this  section,  the  considaed  system  is  asynchronous. 
Each  measurement  arrives  to  the  fusion  centre  with  its 
acquisition  date.  We  note,  tk,  the  acquisition  date  of  the 
current  measurement  and,  tk-i,  the  acquisition  date  of  the 
preceding  measurement.  Each  data  can  be  obtained 
independently  by  the  first  sensor  or  the  second  sensor. 
The  system  takes  the  following  form: 

Dynamic  Xt^  =  F(tk,tk-i)  xtk.i  +  Vtk 

Sensor  i  yl  =  Hi  xtk  +  bt‘k  avec  i  6  {1,2} 

where  ytk  is  sensor  i  measurement  at  time  tk.  F(tk,tk-i) 
is  the  transition  matrix  which  depends  on  the  last 
acquisition  time  and  the  present  time. 

Prediction  equations  are  very  close  to  those  obtained  in 
the  synchronous  case.  The  difference  is  that  a  prediction 
is  realized  each  time  a  measurement  arrives  to  the 
system  and  the  transition  matrix  varies  as  the  time 
difference  between  the  present  acquisition  date  and  the 
preceding  acquisition  date.  With  state  xt^iAk-i 
associated  covariance  Ptn/tk-i.  estimated  at  iteration  k-1, 
prediction  is: 

Xtk/tk-l  ~  F(tk,tk-l)  Xt);.i/tk-i  (1) 

PtkAk-I  ~  F(tk,tk-l)  Ptk-iAk-lP(lk>tk-l)  +  Q(tk>tk-l)  (2) 

where  xt^/ik-i  is  the  predicted  state  at  iteration  k  and  PtkAk-i 
the  associated  covariance.  At  each  time,  the  system  uses 
only  one  measurement.  So,  estimation  equations  are 
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classical  but  depend  on  the  sensor  which  gives  the 
measurement.  If  the  measurement  results  from  sensor  i 
at  time  tk,  the  equations  are  the  following: 

XtkAk  =  xtk/tk-i  +  Ki(k)  (ytk  -  Hi  xtk/tk-i)  (3) 

PtkAk  ~  (I  ■  Hi)  PtkAk-1  (4) 

where  Ki(k)  is  the  Kalman  gain  for  sensor  i 
measurement  and  is  given  by  the  formula: 

Ki(k)  =  P,kAk-iH?'(HiPtkAk-iH?'H-Ri)'‘  (5) 

This  algorithm  is  almost  identical  to  the  sequential 
algorithm  mentioned  above.  The  difference  is  the  use  of 
each  measurement’s  acquisition  dates  in  the  prediction 
process.  Then,  the  state  is  changed  in  taking  the  noise 
characteristics  of  the  sensor  which  delivers  the  current 
measurement  into  account . 


Figure  1  -  Definition  of  a  membership  function 


The  sensor  i  validity  relative  to  the  contextual  variable 
Zj  is  represented  by  the  fuzzy  event  Ci  whose  the 
probability  is  defined  by  the  following  formula  [9]: 

P(Ci)  =  |  pi(zj)p(zj)dzj 

If  the  context  is  identified  with  p  contextual  variables, 
this  expression  becomes  general  and  is  defined  by: 


3.3  Description  of  a  multisensor  tracking 
method  in  taking  context  into  account 


Pi(zi)A..A|Xi(zp)p(zi, ..,  Zp)  dzi..dzp 


If  we  want  to  make  the  performance  of  a  multisensor 
system  optimal,  it’s  necessary  to  study  the  operational 
context  and  to  adjust  to  this  last.  That  is  the  reason  why 
we  propose  in  this  section  a  multisensor  tracking 
method  that  takes  context  into  account.  With  this  aim  in 
view,  the  suggested  algorithm  is  based  on  two  treatment 
levels:  the  first  level  consists  of  filtering  the 
measurements  and  is  almost  identical  to  the 
asynchronous  fusion  algorithm  described  above;  the 
second  level  makes  the  analysis  of  the  context  and 
permits  to  define  a  confidence  coefficient  which  is  taken 
into  account  in  the  first  level  (tracking  system).  This 
coefficient  represents  a  very  simple  way  to  handle  and  a 
very  flexible  tool  to  modulate  the  relative  confidence 
levels  of  two  sources  depending  on  their  own 
characteristics,  their  environment  and  those  of  the 
observed  targets.  This  method  avoids  that  the 
measurements  of  no  reliable  sensors  disturb  the  result  of 
the  global  fusion.  The  objective  is  to  have  a  robust 
system  which  conserves  always  a  track  of  the  target. 

3.3.1  Study  of  the  context 

In  the  second  treatment  level,  it’s  necessary  that  each 
context  in  which  the  system  can  work  is  identified.  A 
particular  context  can  be  defined  by  p  contextual 
parameters,  noted  zj  with  j  g  { 1,  ...,  p}.  Each  contextual 
parameter  permits  to  characterize  the  sensor  validity 
range  with  membership  functions  relative  to  fuzzy 
subsets.  We  give  an  example  of  such  a  function  on 
figure  1.  Here,  the  variable  zi  represents  the  rainfall  rate 
in  ml  by  m^  and  |ii(zi)  the  sensor  1  validity  range:  if 
Hi(zi)=l,  the  sensor  works  nominally,  and  if 
|j,i(zi)  =  0,  the  sensor  doesn’t  work  any  more. 


where  /\  is  the  conjunction  operator  of  fiizzy  subsets. 
Generally,  we  choose  the  min  operator.  If  the 
probability  p(zi, ...,  Zp)  is  identified  to  one  Dirac 
6(z-zi, ...,  z-Zp),  the  probability  P(Ci)  is  comparable  to 
the  conjunction  of  elementary  membership  functions. 
Subsequently,  this  probability  will  be  noted  ft  and  takes 
its  values  in  the  interval  [0,l].  It  measures  the  sensor  i 
reliability  relative  to  identified  context. 

3.3.2  Context  taking  into  account  in  the  fusion  process 
Here,  we  present  the  tracking  algorithm  which  takes 
context  into  account  only  in  the  case  asynchronous.  At 
each  time,  a  sensor  i  measurement  yi  arrives  to  the 
fusion  centre  with  its  acquisition  date  tk  and  its 
confidence  level  ft.  Prediction  is  identical  to  the  one 
defined  by  equations  (1)  et  (2).  The  estimation  results 
firom  a  compromise  between  the  two  following 
alternatives: 

-  the  measurement  is  completely  reliable  and  the 
state  estimation  is  realized  with  equation  (3); 

-  the  measurement  is  not  reliable  and  the  estimated 
state  is  identified  to  the  predicted  state  at  the 
previous  step. 

So,  the  new  estimation  is  given  by  the  formula  [4]: 

Xtk/tk  =  Xtk/tk-i  +  3i  Ki(k)  (ylk  ■  Hi  Xtk/tk-i) 

where  the  Kalman  gain  Ki(k)  is  equivalent  to  the 
preceding  expression  (5).  This  estimated  state  is  obtained 
in  making  the  weighted  average  between  the  two 
previous  states  and  the  associated  covariance  is  lightly 
different  from  the  expression  (4)  because  equal  to: 
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Pti^.  =  (i-Pi)Po  +  3iPi 

with: 

Po  =  PtkAk-1  "  XtkAk-i/v^tkAk  "  ^tkAk-i) 

j  yv  y\  1  ^  /\  1  T 

Pi  =  PtkAk  +  (^tk/tk  "  XtkAkX^tk/tk  ”  Xtk/tk) 
and 

P,U  =  (I-Ki(k)Hi)Pt^.,. 

/si  ^  ^ 

Xtii/tt  =  Xtiytk-i  +  Ki(k)(yt|t  -  Hi  Xtt/tk-i)’ 

We  can  easily  verify  the  two  following  facts:  when  pi  is 
equal  to  1,  the  global  estimate  is  identical  to  the 
estimate  obtained  without  taking  context  into  account; 
when  Pi  is  equal  to  0,  only  prediction  is  taken  into 
account. 

4.  SIMULATIONS 
4.1  Introduction 

This  new  section  suggests  to  validate  the  two  pcceding 
tracking  multisensor  algorithms  which  have  been 
described  in  the  asynchronous  case.  Numerical 
applications  on  which  we  focus  in  this  paper  consist  in 
tracking  a  ground  target  on  the  battlefield  with  a 
multisensor  system  composed  of  two  asynchronous  and 
co-located  sensors.  The  two  systems  which  have  been 
studied  here  are  the  following:  a  system  on  ground 
associating  a  MTI  radar  and  an  IR  camera;  a  system  on 
ground  associating  a  MTI  radar  and  a  TV  camera.  Data 
used  for  algorithms  validation  are  the  following:  a 
terrain  numerical  model  which  contains  hidden  w/and 
forbidden  areas;  trajectory  data  relative  to  a  Active  target 
and  generated  with  a  simulation  tool  develqied  with 
MATLAB. 

Figure  2  gives  terrain  numerical  model’s  representation. 
This  model  is  composed  of  a  river,  three  hidden  areas  and 
a  road  network.  Filtering  is  realized  with  polar 
coordinates  and  uses  two  linear  Kalman  filters  working 
in  parallel  on  range  axis  and  azimuth  axis.  Each  fdter 
considers  a  state  model  with  constant  velocity.  Before 
being  filtered,  the  simulated  measurements  must  be 
changed  and  the  mentioned  transformations  are  the 
following: 

-  position  target  information  are  converted  in  polar 
coordinates; 

-  we  add  to  different  measurements  an  observation 
noise  which  characterizes  the  considered  sensor. 

Subsequently,  we  suggest  to  adopt  the  following 
notations: 

Ai  and  A2  are  respectively  the  sampling  periods  of 
sensors  in  seconds; 

Dt  is  the  treatment  time  in  seconds; 

As  is  the  wished  output  period; 


Odi  is  the  noise  standard  deviation  relative  to  radar  range 
(in  meters); 

Oai  is  the  noise  standard  deviation  relative  to  radar 
azimuth  (in  radians); 

Oa2  is  the  noise  standard  deviation  relative  to  IR  camera 
azimuth  (in  radians); 

The  performances  of  the  algorithms  are  evaluated  with 
the  mean  absolute  error.  This  last  one  is  designed  by 
eniixy  and  is  the  difference  between  the  true  trajectory  of 
the  target  and  the  estimated  one.  We’ll  design  by  erm; 
the  mean  absolute  error  relative  to  range  filter  and  by 
ermj  the  one  relative  to  azimuth  filter. 

4.2  Numerical  application  1 
The  first  battlefield  scenario  on  which  we  focus  is 
represented  on  figure  3.  It’s  composed  of  a  target  whose 
trajectory  is  rectilinear  at  the  beginning  and  which  takes 
a  bend  to  90°  at  a  given  time  on  several  seconds.  The 
target  moves  behind  hidden  zones.  We  suppose  the  MTI 
radar  is  only  disturbed  by  areas  presence. 

The  sensor  characteristics  are  known  and  defined  by: 

Ai  =  4  s,  Odi  =  40  m  and  Oai  =  3.10'^  rd  for  the  MTI 
radar; 

A2  =  2  s  and  aa2  =  10'^  rd  for  the  IR  camera. 

Concerning  the  state  noise  variance,  we’ll  take  at  each 
test  oil  =  0.1  for  the  first  filter,  and  0^2  =  0.1  for  the 
second  filter.  We’ll  suppose  also  that: 
Dt  =  800  s  and  As  =  2  s  for  each  Mter.  Radar  disturbance 
relative  to  hidden  areas  is  simulated  with  an  additional 
Gaussian  noise  which  is  defined  by  parameters: 

Cdci  =  320  m  for  range, 

and  Oaci  =  0.0245  rd  for  azimuth  angle, 

and  which  is  added  to  radar  measurements.  In  the 
classical  tracking  algorithm,  all  measurements  are 
filtered  but  in  the  second  algorithm,  we  take  context  into 
account  with  the  confidence  level  p  which  can  take  the 
two  following  values: 

-  P  =  0  if  radar  measurement  isn’t  reliable; 

-  p  =  1  if  radar  measurement  is  reliable. 

In  the  first  case,  state  isn’t  updated  with  radar 
measurement  and  in  the  second  case,  it  is  updated. 

Table  1  gives  different  absolute  error  rates  obtained  from 
the  two  algorithms.  Case  1  is  relative  to  the  first 
algorithm  test  and  case  2  to  the  second  algorithm  test. 
This  table  shows  that  the  second  method  gives  the  best 
performance.  It’s  obvious  that  the  integration  of 
contextual  information  permits  to  improve  effectively 
the  target  tracking.  Even  if  the  first  algorithm  yields 
worse  results  than  the  second  method,  the  obtained 
precision  in  the  case  1  is  nevertheless  very  acceptable. 
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ermxy(m) 

ermi(m) 

erm2(rd) 

case  1 

32.914 

41.686 

9.67.  lO"* 

case  2 

17.919 

20.205 

7.96.  10’’' 

Table  1  -  Mean  absolute  errors  obtained  from  the  two 
algorithms 

Figure  4  presents  the  variation  of  parameter  p  according 
to  time.  We  see  that  this  coefficient  is  zero  when  the 
radar  measurements  are  perturbed  by  the  hidden  areas  and 
is  equal  to  one  in  the  other  case.  Finally,  figure  5  is 
composed  of  two  different  curves:  the  first  one  is 
designed  by  a  dotted  line  and  gives  the  absolute  erro- 
variation  according  to  time,  obtained  fiom  the  classical 
method;  the  second  one  is  described  by  a  continuous  line 
and  presents  the  absolute  error  variation  obtained  when 
we  take  context  into  account.  The  comparison  of  the 
two  curves  on  this  last  figure  reveals  that  the  difierence 
between  the  trae  trajectory  and  the  estimated  trajectory  is 
on  average  slighter  with  the  second  method  than  with 
the  first  one.  Consequently,  we  can  say  that  the  tracking 
algorithm  in  taking  context  into  account  is  more 
satisfactory  than  the  classical  tracking  algorithm. 

4.3  Numerical  application  2 

The  second  simulation  is  relative  to  battlefield  scenario 
which  is  presented  on  figure  6.  This  last  is  composed  of 
a  target  whose  trajectory  is  rectilinear.  The  main  aim  of 
this  application  is  to  compare  the  two  tracking  methods 
in  validating  the  second  algorithm  with  a  variable 
context  coefficient  p  which  can  take  any  value  in  the 
interval  [0,l]  and  in  using  the  Monte  Carlo  method.  The 
contextual  variable  on  which  we  focus  here  can 
characterize  for  example  a  jamming  radar  level.  We 
suppose  that  the  two  sensors  have  a  sampling  period 
equal  to  4  seconds,  the  wished  output  period  is 
equivalent  to  4  seconds  and,  finally,  the  treatment 
duration  is  equal  to  400  seconds.  We  know  also  that 
precision  of  different  sensors  is  defined  by: 

Odi  =  40  m  and  Oai  =  3.10’^  rd  for  the  MTl  radar; 
aa2  =  0.5.10'^  rd  for  the  ER  camera. 

WeTl  fix  at  each  test  the  state  noise  variance  at  ch  =  1 
for  the  first  filter,  and  oh  =  lO  ’  for  the  second  one. 
Figure  7  gives  the  variation  of  coefficient  p  according 
to  time  used  at  each  test.  In  other  respects,  when  radar 
measurements  are  not  reliable,  we  add  to  them  an 
additional  Gaussian  noise  which  is  characteristic  of  the 
disturbance  importance.  The  two  tracking  algorithms  are 


canpared  with  the  Monte  Carlo  method  in  using  100 
different  realizations  of  measurement  noise.  The  mean 
result,  which  is  relative  to  absolute  error  and  obtained 
for  each  algorithm  on  all  realizations,  is  the  following: 

Method  1  - >  OTtixy  =  64.963  m 

Method  2  - >  eririxy  =  44.378  m  . 

On  figure  8,  we  compare  the  absolute  errors  which  are 
obtained  from  the  two  methods  for  one  particular 
realization  of  measurement  noise. 

As  in  the  first  application,  we  observe  again  that  the 
tracking  algorithm  taking  context  into  account  is  the 
one  which  gives  the  best  performance  on  average.  We 
can  say  that  the  using  of  contextual  information  in 
tracking  involves  to  eliminate  unreliable  radar 
measurements  in  the  filtering  and  permits  to  have  a 
robust  tracking  system. 

5.  CONCLUSION 

In  this  paper,  two  multisensor  tracking  algorithms  have 
been  studied  and  validated  for  battlefield  applications. 
The  first  algorithm  is  quite  classical  but  the  second 
algorithm  is  the  more  interesting  method  because  it 
takes  context  into  account  in  tracking.  Several  numerical 
simulations  based  on  realistic  battlefield  scenarios  have 
been  described  and  have  permited  to  make  a  comparative 
study  of  the  two  proposed  algorithms.  This  analysis 
demonstrates  that  the  algorithm  taking  context  into 
account  gives  the  best  performance  and  yields  to  a  very 
robust  multisensor  tracking  system.  An  other  advantage 
of  this  method  is  that  it’s  easy  to  implement  because  it 
involves  only  to  make  the  analysis  of  the  operational 
context  and  to  define  a  fuzzy  confidence  coefficient 
which  is  integrated  in  the  tracking  system.  This 
parameter  is  a  simple  way  to  handle  and  to  modulate  the 
relative  confidence  levels  of  two  sensors  depending  on 
their  own  characteristics,  their  environment  and  those  of 
the  observed  target.  Several  further  steps  of  work  might 
be  craisidered:  taking  other  contextual  information 
sources  into  account  as  for  example  weather  forecast, 
vegetation  or  jamming;  global  fusion  of  several  not 
located  sub-systems  (association  of  sensors)  fw 
determining  contribution  of  such  an  architecture  feced 
with  the  masking  problem;  study  of  multitarget 
scenarios  and  finally,  taking  false  alarms  into  account  in 
the  multisensor  tracking. 
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Figure  6  -  Scenario  2  of  battlefield 


Figure  7  -  Variation  of  coefficient  P  according  to  time  used  in  application  2 
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Figure  8  -  Absolute  error  of  estimated  trajectory  in  application  2 
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Summary 

New  imaging  sensors  and  high  performance  computer 
graphics  systems  have  a  great  impact  on  the  design  of 
the  human  machine  interface  of  novel  cockpit  sys¬ 
tems.  The  improvement  of  aircraft  safety  and  opera¬ 
tional  qualities  under  adverse  weather  can  be 
achieved  by  so-called  enhanced  vision  systems,  which 
consist  in  principle  of  a  combination  of  sensor-  and 
synthetic-vision.  The  following  contribution  describes 
the  enhanced  vision  concept  which  has  been  devel¬ 
oped  at  the  Institute  of  Flight  Guidance  of  the  Ger¬ 
man  Aerospace  Center  (DLR).  Benefits  of  competing 
imaging  sensors  are  compared,  results  of  recently 
conducted  field  tests  are  shown,  and  first  concepts 
for  a  fusion  of  radar  and  synthetic  images  are  pre¬ 
sented. 


1.  Introduction 

Actual  investigations  on  new  cockpit  technologies  are 
focused  on  improving  the  situational  awareness  of  the 
cockpit  crew,  especially  during  approach,  landing, 
take-off  and  taxiing.  The  role  of  this  key  element  is 
emphasized  by  the  following  quotation  of  the  Human 
Factors  Team  (SA-1)  of  the  FAA: 

“The  FAA  should  require  operators  to  increase 
flightcrews  understanding  of  and  sensitivity  to 
maintaining  situation  awareness,  particuiariy: 
...  position  awareness  with  respect  to  the  intended 
flight  path  and  proximity  to  terrain,  obstacles  or 
traffic."  ■ 

Particularly  future  military  transport  aircraft  have  to 
cope  with  new  requirements  such  as  autonomous, 
non-cooperative  landing  at  unsupported  (without  D- 
GPS,  MLS,  ILS)  airstrips  down  to  CAT-1  (minimum)  or 
better,  low  level  flight  operations,  ground  mapping, 
precise  air  dropping,  search  and  rescue  missions.  In 
most  cases  these  requirements  have  to  be  established 
under  adverse  weather  conditions,  without  being 
detected  by  hostile  observation. 

Within  this  context  visual  information  provided  by 
fusion  of  image  data  from  onboard  multispectral  sen¬ 
sors  with  synthetic  vision,  supported  by  an  ATC- 
interface  and  aircraft  state  data  (position,  attitude, 
speed,  etc.),  will  become  an  important  and  helpful 
tool  for  aircraft  guidance.  The  development  of  these 


so  called  "Enhanced  Vision  Systems"  (EVS)  is  an  inter¬ 
disciplinary  task,  which  requires  a  wide  spectrum  of 
different  information  technologies: 

•  modern  data  link  technology  for  transmission  of 
guidance  information; 

•  complex  data  bases  to  provide  terrain  data  and 
high  performance  computer  graphics  systems  to 
render  synthetic  images  in  real  time; 

•  a  new  generation  of  onboard  imaging  sensors,  like 
solid  state  infrared  and  especially  a  new  kind  of 
imaging  radar,  providing  a  real  view  through  dark¬ 
ness  and  adverse  weather; 

•  knowledge  based  image  interpreters  to  convert 
sensor  images  into  a  symbolic  description. 

This  paper  presents  the  DLR  concept  for  an  integrated 
enhanced  vision  system.  After  the  description  of  the 
basics  in  section  2  the  image  sensor  characteristics  are 
compared  with  the  requirements  of  an  Enhanced 
Vision  System  in  section  3.  First  results  concerning  the 
experiments  with  the  DASA  HiVision  radar  and  data 
fusion  techniques  are  given  in  section  4. 


Sensor 

Vision 

•  no  single  sensor  will  really  cover  all 
possible  weather  situations 

•  what  happens  if  the  imaging  sensor 
fails? 

•  multispectral  images  are  difficult  to 
interpret 

Synthetic 

Vision 

•  unmodelled  obstacles  are  not  detect¬ 
able 

•  how  can  the  integrity  of  the  data  base 
be  monitored  and  what  happens  if 
data  base  errors  occur? 

•  how  can  such  a  data  base  be  certified? 

•  position  accuracy  and  reliability  of  the 
synthetic  image  depends  on  the  accu¬ 
racy  and  reliability  of  the  reference  sys¬ 
tem. 

Table  2.1:  Problems  of  Sensor  Vision  and  Synthetic  Vision 
as  isolated  technologies. 


'  This  work  was  sponsored  by  the  German  Ministry  of  Defense  under  the  title:  "Bildverarbeitung  zur  Pilotenunterstutzung" 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-I2. 
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2.  The  DLR  Concept  for  Enhanced  Vision 

Looking  a  little  closer  to  the  meaning  of  the  terms 
"Enhanced  Vision  Systems"  (EVS)  and  "Synthetic  Vi¬ 
sion  System"  (SVS),  we  find  a  rather  indistinct  situa¬ 
tion:  sometimes  the  whole  field  concerning  "obstacle 
detection",  "sensor  simulation",  "4D  flight  guidance 
display",  "enhanced  vision",  etc.  is  subsumed  under 
the  headline  “Synthetic  Vision"  [20,8]  and  sometimes 
the  same  term  is  strictly  reduced  to  computer  gener¬ 
ated  images  [17], 

It's  amazing  to  see  that  the  terms  “EVS"  and  "SVS" 
are  very  often  discussed  in  parallel,  -  sometimes  even 
treated  as  competing  concepts.  But  as  long  as  EVS 
and  SVS  are  considered  as  isolated  technologies,  they 
have  to  face  some  critical  questions,  such  as  shown  in 
Table  2.1. 

Another  somewhat  restricted  but  quite  interesting 
definition  concerning  the  field  of  EVS-,  SVS- 
terminology  was  coined  by  the  Air  Transport  Associa¬ 
tion  (AT A) : 

"...  means  to  safely  increase  airport  capacity  and 

reduce  runway  incursions  in  low  visibility  condi¬ 


tions,  without  significant  expansion  of  ground  fa¬ 
cilities.  " 

This  definition  seems  to  be  mainly  influenced  by  civil 
needs,  but  the  renunciation  of  supporting  ground 
facilities  makes  EVS-  and  SVS-technology  also  very 
interesting  for  military  requirements. 

To  avoid  the  drawbacks  of  the  isolated  EVS  and  SVS 
technologies  and  to  maintain  the  benefits  of  both,  it 
seems  to  be  absolutely  necessary  to  integrate  them 
into  one  system  which  should  be  treated  as  a  part  of 
an  artificial  pilot  assistant. 

In  order  to  avoid  any  confusion  concerning  the  terms 
used  throughout  this  paper,  it  seems  to  be  worth¬ 
while  to  define  them  : 

"Sensor  Vision"  =  Sensor  generated  images;  (replaces 
the  term  "Enhanced  Vision"  in  the  conventional 
meaning). 

"Synthetic  Vision"  =  Computer  generated  images 
based  on  navigation  inputs,  map  information  and/or 
terrain  data  bases. 

"Enhanced  Vision"  =  A  concept  which  integrates 
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Fig.  2.1  :  Block  Diagram  of  the  DLR  "Integrated  Enhanced  Vision  System". 
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Advantage 

Disadvantage 

Synthetic  Vision 

easy  image  interpretation 

no  obstacle  detection 
reference  system  necessary 
depends  on  data  base  reliability 

+  Sensor  Vision 

obstacle  detection 
no  reference  system  necessary 
no  data  base  necessary 

difficult  image  interpretation 

+  Aircraft  status 

already  available 

single  sensor  information 

+  ATC  data 

identification  of  other  aircraft 

needs  data  link  technology 

=  Integrated  Enhanced  Vision  System 

Table  2.2:  Definition  of  the  Integrated  Enhanced  Vision  System. 


"Sensor  Vision",  "Synthetic  Vision",  aircraft  state- 
and  ATC-data. 

Figure  2.1  shows  the  basic  idea  of  an  integrated  En¬ 
hanced  Vision  concept  [7,12].  Key  elements  are  func¬ 
tional  subblocks  which  are  interacting  through  de¬ 
fined  interfaces:  The  "Sensor  Vision"  block  includes 
the  imaging  sensor(s)  and  possibly  some  image  data 
preprocessing.  The  "Vision  Processing"  block  is  re¬ 
sponsible  for  a  image  processing  concerning  signal 
restoration  like  noise  reduction  and  various  filtering 
techniques,  scaling  and  transformation  to  convenient 
perspectives  and  as  a  most  challenging  topic:  feature- 
or  object  extraction.  The  block  "Synthetic  Vision" 
represents  all  necessary  technologies  to  set  up  a  syn¬ 
thetic  image  derived  from  terrain  data  bases  and 
navigational  information. 

The  core  part  of  the  integrated  Enhanced  Vision  Sys¬ 
tem  is  represented  by  the  block  "Vision  Fusion".  A 
first  and  simple  possibility  for  "Vision  Fusion"  could 
be  an  integration  of  sensor  images  and  flight  status 
data  on  the  primary  flight  display,  using  a  Head  Down 
Display  (HDD)  or  a  Head  Up  Display  (HUD)  in  order  to 
increase  the  crew's  situation  awareness,  but  other 
possibilities  are  just  as  important: 

•  incompatibilities  between  sensed  data  and  terrain 
data  can  be  used  for  obstacle  detection; 

•  a  conformity  check  between  sensed  vision  and 
synthetic  vision  can  be  used  as  an  "Integrity  Moni¬ 
tor"  for  the  navigation  system  and  the  data  bases. 

This  concept  should  be  regarded  as  part  of  a  pilot 
assistant  system,  which  will  not  be  discussed  in  this 
paper.  But  it  is  obvious  that  obstacle  detection  carried 
out  within  the  "Vision  Fusion"  subblock  can  be  im¬ 
proved  to  an  obstacle  or  vehicle  identification,  if  ATC 
information  is  included.  The  pilot  assistant  may  con¬ 
dense  these  data  to  a  representation  of  the  traffic 
situation  and  -  if  the  need  arises  -  to  a  conflict  predic¬ 
tion. 

Table  2.2  summarizes  the  concept  of  an  integrated 
Enhanced  Vision  System.  It  is  obvious  that  the  disad¬ 
vantages  of  "Synthetic  Vision"  and  "Sensor  Vision" 


cancel  out  each  other  and  the  advantages  are  in¬ 
creased.  In  the  sense  of  a  synergetic  effect  both  sum 
up  to  a  new  system  quality  in  the  fields  of: 

Crew  Assistance  in  connection  with: 

•  adverse  weather, 

•  obstacle  detection, 

•  non-cooperative  landing  systems. 

Integrity  Monitor  for: 

•  navigation, 

•  GPWS, 

•  taxi  guidance  system. 

Automatic  Flight  Guidance  in  terms  of : 

•  image  based  navigation, 

•  collision  avoidance, 

•  avoidance  of  terrain  and  CFIT. 


3.  Imaging  Sensors  -  Characteristics  and  Re¬ 
quirements 

The  requirements  for  suitable  imaging  EVS  sensors 
depend  strongly  on  the  application.  In  the  following 
table  the  civil  requirements  and  the  resulting  sensor 
characteristics  should  be  interpreted  as  the  minimum 
performance  and  the  military  features  are  meant  as  an 
add  on. 

With  the  background  of  the  requirements  shown  in 
table  3.1  we  should  be  able  to  qualify  different  types 
of  sensors  in  terms  of  their  EVS  capability.  A  rather 
convenient  method  to  distinguish  between  all  possible 
sensor  types,  is  to  refer  to  the  operating  wavelength, 
because  this  parameter  allows  a  rather  easy  estima¬ 
tion  for  two  of  the  most  important  sensor  parameters: 
the  penetration  through  the  atmosphere  (especially 
under  foggy  conditions)  and  the  image  quality  in 
terms  of  resolution.  The  rule  of  thumb  is  rather  sim¬ 
ple:  increasing  wavelength  improves  the  penetration 
and  decreases  the  image  resolution.  But  of  course 
there  is  no  rule  without  exception:  if  the  wavelength 
is  much  smaller  than  the  water  particle  size  (1-10 
microns),  the  penetration  increases  again.  This  effect 
is  used  for  the  so  called  "FogEye"  -  receiver,  which  is 
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ooerational  requirements 

resultinq  requirements  for  imaqinq  sensors 

civil 

•  reduction  of  minimum  approach  RVR 

•  reduction  of  minimum  takeoff  RVR 

•  safe  taxi  operations 

•  obstacle  warning  in  critical  phases 

•  CFIT  warning 

•  integrity  monitor  for  GPS  supported  naviga¬ 
tion 

•  search  and  rescue 

=»  weather  independent 
^  sufficient  resolution 
^  image  rate  >  1 5  Hz 
image  delay  <  200  ms 
=>  coverage  minimum  =  HUD  coverage 

military 

•  low  level  flights 

•  landing  on  unsupported  forward  operating 
strips 

•  precise  air  dropping 

•  air  surveillance 

•  ground  mapping  for  surveillance 

passive  or  "silent"  sensor 
autonomous;  ground  facilities  not  necessary 
=>  reconnaissance  capability 

Table  3.1:  Requirements  for  imaging  EVS-sensors  for  civil  and  military  applications.  (RVR=Runway  Visual  Range; 
CFIT=Controlled  Flight  Into  Terrain) 


optimized  for  UV  wavelengths  between  0.2-0.275 
microns  [9].  Table  3.2  compares  the  characteristics  of 
sensors  which  might  come  into  consideration  for  EVS 
applications. 

A  valid  comparison  of  the  all-weather  capability  is 
nearly  impossible,  because  the  technical  realizations  of 
image  generation  are  too  different.  First  of  all,  it 
seems  quite  clear  that  the  active  kind  of  sensors  are 
more  successful  penetrating  a  foggy  or  rainy  atmos¬ 
phere  than  the  passive  ones.  "Active"  in  this  context 
is  not  only  an  illuminating  kind  of  device,  such  as  the 
active  radars  (LADAR,  MMW,  PBMMW),  but  also 
those  sensors  which  need  an  emitting  source  to  over¬ 
come  the  signal  to  noise  barrier,  as  for  instant  the  UV 
sensor  "Fog  Eye"  [9]  and  to  a  certain  extent  the  "pas¬ 
sive  "-MMW  (PMMW)  camera,  whose  range  of  visibil¬ 
ity  can  be  significantly  enlarged,  if  additional  MMW- 
reflectors  are  positioned  nearby  the  target.  [14].  If  we 
take  operational  requirements  into  account  we  can 
define: 

Active  sensor  systems  make  use  of  additional 
means  to  increase  the  emission  at  or  nearby  the 
observed  object. 

From  this  point  of  view  the  UV-,  LADAR-,  MMW-  and 
PBMMW-sensor  are  purely  active,  the  others  are  op¬ 
tional  active  depending  on  the  usage  of  illumination 
or  emission  increasing  devices,  such  as  IR-emitting 
lamps  or  MMW-reflectors,  etc. 

On  the  other  hand  a  "passive"  sensor  is  not  necessar¬ 
ily  the  same  as  a  "silent"  one.  For  example:  the  active 
MMW-sensor,  as  it  is  proposed  by  DASA  Ulm  [11], 
has  a  greater  range  than  the  hostile  ESM  detection 
range.  This  is  an  example  for  an  active  but  silent  sen¬ 
sor;  and  it  turns  out  that  the  meaning  of  term  "silent" 
is  a  matter  of  mission. 


But  let's  return  to  the  "all  weather"  capability:  A 
rather  convincing  method  to  define  the  "all  weather" 
characteristic  of  a  sensor  was  proposed  by  W.F.  Horne 
et.al.  [8].  Here  the  difference  of  received  power  from 
the  runway  and  the  surrounding  grass  as  a  function  of 
distance  and  various  weather  conditions  had  been 
measured.  The  variation  of  the  received  power  be¬ 
tween  grass  and  runway  was  used  as  a  measure  for 
the  contrast  and  a  3dB  difference  was  defined  as 
threshold  for  sufficient  visibility.  The  results  of  these 
investigations  for  a  35GHz  MMW-sensor  and  an  IR- 
sensor  is  given  in  table  3.2. 

Although  the  visibility  data  of  the  other  types  of  sen¬ 
sors  are  not  completely  comparable  we  can  state,  that 
the  MMW-sensor  technology  seems  to  be  the  most 
successful  one. 

Another  very  important  feature  of  the  above  listed 
sensors  is  the  type  of  imaging:  UV-,  IR-,  Video-  and 
PMMW-sensors  generate  a  perspective  2D  kind  of 
image,  which  the  human  visual-perception-system  is 
evolutionary  trained  to  process  into  a  3-D- 
representation  of  the  "outside  world".  The  MMW- 
sensor  on  the  other  hand  delivers  primarily  an  infor¬ 
mation  about  the  range  and  the  angular  direction  of  a 
certain  object.  This  range-angle  information  can  be 
transformed  into  a  view  "out-of-the-window",  but 
there  is  still  a  lack  of  information  about  the  objects 
height  or  its  vertical  position  [5].  The  presentation  of 
such  images  needs  knowledge  about  the  surrounding 
elevation,  which  often  is  estimated  by  the  so-called 
"flat-earth-assumption "[10].  On  the  other  side  a 
MMW-sensor  delivers  a  direct  information  about  the 
distance  to  certain  objects,  which  is  extremely  valu¬ 
able  for  navigation,  traffic  analysis  and  obstacle  detec¬ 
tion  and  obstacle  avoidance  respectively. 
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sensor 

type 

wave¬ 

length 

[micron] 

kind  of 
image 

active  / 
passive 

ground 

facility 

angular 

resolution 

[degree] 

range 

resolution 

[m] 

rate 

[Hz] 

delay 

[sec] 

FOV 

HxV 

[degree] 

visibility 

range 

[km] 

UV 

[9] 

0.2-0.3 

2-D 

optical 

yes’) 

0.05 

^) 

■ 

■ 

30x22 

0.8 

Video 

0.4-0.8 

2-D 

optical 

passive 

no 

0.05'’) 

') 

25 

■ 

variable 

■ 

LADAR 

[21] 

1.54 

2.5-D 

range- 

angle- 

anqle 

active 

no 

0.35x0.20 

1 

■ 

32x32 
range  1  km 

■ 

IR 

[8] 

3-5 

a 

passive 

_ 

no 

0.05”) 

') 

25 

■ 

variable 

■ 

IR 

8-12 

a 

no 

0.05'’) 

') 

25 

■ 

variable 

■ 

MMW 

94GHz 

[8,3] 

3190 

2-D 

range- 

angle 

active 

(no) 

“) 

3 

10 

■ 

30x22 

■ 

MMW 

77GHz 

[4] 

3900 

variable 

=) 

active 

(no) 

0.25 

1.2 

■ 

■ 

■ 

MMW 

77GHz 

[13] 

3900 

variable 

active 

1.00 

1 

■ 

■ 

■ 

MMW 

35GHz 

[8,11] 

8570 

2-D 

range- 

angle 

active 

(no) 

0.25-0.80 

') 

6 

16 

<0.2 

40x28 
range  3- 
10km 

3.0 

■TjQJivllll 

WKmaM 

8570 

2-D 

range- 

angle 

active 

(no) 

0.70 

7-15 

10.5 

<0.2 

3.0 

PBMMW 

35GHz 

[1] 

8570 

2.5-D 

range-ar 

gle- 

anqle 

active 

no 

2.5 

5-10 

0.5-1 .0 

1-2 

50x20 

range  1-3km 

3.0 

PMMW 

[14,2,15] 

3190-8570 

2-D 

optical 

passive 

(no) 

0.15-0.50 

0 

17 

■ 

15x17 

0.7 

Table  3.2:  Characteristics  of  potential  EVS-sensors  under  adverse  weather. 


UV 

Ultra  Violet 

LADAR 

LAser  raDAR 

IR 

Infra  Red 

MMW 

MilliMeter  Wave  radar 

PBMMW 

Pencil  Beam  MilliMeter  Wave  radar 

PMMW 

Passive  MilliMeter  Wave  sensor 

')  UV-source  on  the  ground  necessary, 

')  direct  range  information  not  available, 

1m  antenna  square  aperture  [2] 

“)  calculated  for  field  of  view  (FOV)  of  40° 

*)  specific  values  are  not  available,  but  problems  are  not  expected, 
data  not  available, 

’)  numerical  resolution  0.25°,  beamwidth:  0.8°, 

“)  no  improvement, 

’)  depends  on  scanning  method. 
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UV 

Video 

IR 

LADAR 

MMW 

PBMMW 

PM  MW 

weather  independent 

• 

- 

- 

- 

+ 

+ 

• 

sufficient  resolution 

• 

+ 

+ 

+ 

- 

- 

- 

imaqe  rate  >  1 5  Hz 

+ 

+ 

+ 

- 

+ 

- 

+ 

image  delay  <  200  ms 

+ 

+ 

+ 

- 

+ 

- 

7 

coverage  (azimuth  x  elevation) 

+ 

+ 

+ 

+ 

+ 

+ 

• 

passive  sensor 

yes 

yes 

yes 

no 

no 

no 

yes 

"silent"  sensor 

no/yes 

yes 

yes 

no 

yes 

no 

yes 

autonomous 

no 

yes 

yes 

yes 

yes 

yes 

yes/no 

reconnaissance  capability 

no/yes 

yes 

yes 

yes 

yes 

no/yes 

yes 

Table  3.3:  Matrix  of  EVS-requirements  and  sensor  characteristics  (+:  good  ;  •;  fair;  poor). 


But  whatever  conclusions  are  drawn  from  these  dif¬ 
ferent  kinds  of  image  generation,  it  seems  to  be  im¬ 
possible  to  claim  a  general  advantage  for  one  of 
them.  In  connection  with  the  above  used  definition  of 
the  term  "Enhanced  Vision"  as  a  fusion  of  "Synthetic 
Vision"  and  "Sensor  Vision",  the  type  of  imaging  is  an 
important  part  of  the  system  philosophy. 

On  the  basis  of  the  sensor  requirements  (Table  3.1) 
and  the  sensor  characteristics  (Table  3.2)  we  should 
be  able  to  correlate  sensor  characteristics  and  re¬ 
quirements  (Table  3.3). 

Simply  counting  the  "+"  and  "yes".  Table  3.3  gives 
the  impression,  that  the  infrared  sensors  and  the  sim¬ 
ple  video  cameras  should  be  the  most  promising  sen¬ 
sors  for  EVS  applications.  They  offer  nearly  all  fea¬ 
tures,  which  might  be  valuable,  except  one:  the  ability 
to  penetrate  fog,  snow  and  rain. 

Horne  et.al.  [8]  stated  in  their  paper: 

"During  approaches  and  landings  in  actual 
weather  for  50  foot  ceilings  and  700-1000  foot 
visibility  the  millimeter  wave  image  was  not  no¬ 
ticeable  degraded.  During  these  same  in-weather 
approaches,  the  IR  sensor  was  not  able  to  provide 
an  image  any  better  than  the  human  eye. " 

If  EVS-technology  is  mainly  justified  by  an  increase  of 
the  crew’s  (visual)  situation  awareness  under  adverse 
weather  conditions  the  all  weather  capabilities  of  the 
sensors  will  become  the  most  important  characteristic. 
From  this  point  of  view  the  UV-,  the  MMW-  and  the 
PMMW-sensor  should  be  taken  into  account.  The  UV- 
sensor  and  the  passive  MMW-sensor  need  some  addi¬ 
tional  ground  facilities  (UV-sources  and  MMW- 
reflectors),  the  first  one  in  general  and  the  latter  one 
in  adverse  weather  conditions.  [9,14],  Additional 
ground  facilities  might  be  no  problem,  especially  if 
they  are  cheap  and  easy  to  install,  but  they  restrict  the 
EVS  technology  to  certain  scenarios  with  a  ground 
based  infra  structure,  which  might  be  not  always 
available,  especially  for  military  applications. 

These  are  the  reasons  why  DLR  decided  to  use  the 
"HiVision"  MMW-radar  from  DASA  Dim  as  the  main 
sensor  for  EVS  research.  Technical  descriptions  are 
given  in  several  papers  [11,19];  therefore  a  short 


summary  might  be  sufficient:  "HiVision"  radar  uses 
continous  wave  (FMCW)  technology  with  a  frequency 
scanning  antenna  which  covers  an  azimuth  sector  of 
40°.  Table  3.4  summarizes  the  parameters  which  were 
achieved  during  recent  tests. 

A  very  promising  feature  of  this  sensor,  especially  for 
military  applications,  is  the  ESM  range,  where  the 
radar  could  be  detected.  With  an  output  power  of 
lOOmW/BOOmW  a  high  performance  ESM  receiver 
with  a  -80dBm  sensitivity  would  detect  the  radar 


waveform 

FMCW 

scanning  principle 

frequency  scanning 

centre  frequency 

35  GHz 

scanning  (imaqe)  rate 

16  Hz 

emitted  power 

500  mW 

resolution  (angle) 

0.26°  (157  pixel/41  °) 

azimuth  beamwidth 

0.8° 

azimuth  coverage 

41° 

range  resolution 

6  m  (512  pixel/3300m) 

Instrumented  range 

3.5  km 

detection  range  against  a 
person 

10  km 

weight  (radar  head) 

approx.  1 5  kg 

antenna  size 

86x1 5x30  cm" 

Table  3.4:  Characteristics  of  the  DASA  "HiVision"  radar. 


within  a  distance  of  2. 3/5. 2km.  On  the  other  hand: 
the  radar  range  against  a  person  (radar  cross  section 
approx.  ImO  is  6.9/1 0km,  which  means  that  the  hos¬ 
tile  detection  range  is  much  smaller  than  the  detec¬ 
tion  range  of  the  radar  itself  [11],  Or  in  other  words: 
the  aircraft  with  an  active  HiVision  radar  can  be  de¬ 
tected  earlier  by  "ears"  than  by  an  ESM  receiver. 


4.  Results 

The  following  chapter  describes  results  concering 
driving  trials  and  flight  tests  and  some  aspects  of  ra¬ 
dar  image  processing  and  fusion. 


Fig.  4.1  :  Multi  sensor  test  van,  equipped  with  the  MMW-radar  Fig.  4.2:  DASA  HiVision  radar  mounted  on  top  of  a  DO-228, 
(aluminium  box  on  the  top),  a  video-  and  an  IR-sensor. 


4.1  Experimental  Setup 

The  DASA  HiVision  radar  is  a  rather  new  development 
which  was  first  investigated  within  a  static  environ¬ 
ment  [19],  The  Institute  of  Flight  Guidance  at  DLR 
Braunschweig  started  to  take  the  first  step  from  static 
scenario  to  a  dynamic  one  in  1996.  In  order  to  estab¬ 


lish  a  rather  quick  experience,  a  Mercedes  Benz  van 
(Fig.  4.1)  was  equipped  with  a  differential  GPS  re¬ 
ceiver,  with  a  Litton  inertial  reference  unit  (to  deter¬ 
mine  the  exact  trajectory  during  the  test)  and  a  set  of 
forward  looking  imaging  sensors  for  the  visible  and 
the  infrared  channel.  The  visible  channel  was  covered 


Fig.  4.3:  A  set  of  typical  multispectral  images  of  a  Braunschweig  airport  taxi  way.  a)  visible  channel,  b)  IR  channel, 
c)  Radar  "out-the-window-view",  d)  Radar  map  view. 
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Fig.  4.4  ;  C-scope  radar  image  with  "obstacle". 

with  a  video  camera  type  FA  871  (0,4-1 .0  micron 
wavelength)  and  a  resolution  of  581x756  pixel  manu¬ 
factured  by  Grundig.  For  the  infrared  channel  the 
AEGAIS  -  camera  manufactured  by  AIM  Heilbronn 
(former  AEG)  for  the  3-5  micron  wavelength  and  a 
spatial  resolution  of  256x256  pixel  was  used.  A  set  of 
typical  images  are  shown  in  Fig.  4.3. 

In  parallel  DASA  and  DLR  developed  an  airborne  ver¬ 
sion  of  the  HiVision  Radar  for  the  DO  228  (Fig.  4.2)  , 
First  flight  tests  are  planned  for  the  autumn  of  1998. 

To  avoid  any  misunderstanding:  the  "AWACS-like" 
position  of  the  radar  on  top  of  the  aircraft  (Fig.  4.2) 
was  chosen  to  avoid  any  kind  of  disturbance  between 
the  already  existing  IR-sensor  at  the  nose  of  the  plane 
and  the  radar.  With  a  width  of  the  radar  antenna  of 
approx.  80cm  it  should  be  always  possible  to  find  a 
position  inside  the  radom  at  the  aircraft's  nose,  at 
least  at  medium  sized  or  large  transport  aircraft. 


4.2  Experiments  with  DASA  HiVision  Radar 

Sensor  data  sets  were  recorded  on  Braunschweig 
airfield  driving  a  well  defined  course  along  taxi-  and 
runways.  The  experiments  were  repeated  at  different 
times  of  the  day  (even  at  night)  and  under  different 
weather  conditions  ,  such  as  sunny  and  cloudy  sky, 
rain  and  fog.  Furthermore  some  artificial  obstacles  like 
other  vans  were  placed  on  the  track  and  natural  ob¬ 
stacles  (Fig.  4.4)  appeared  by  random. 

Fig.  4.3  shows  a  set  of  images  acquired  during  a  typi¬ 
cal  test  run  looking  along  a  concrete  runway  towards 
an  area  with  parking  airplanes.  While  Fig.  4.3  a)  and 
b)  show  unprocessed  images  of  the  visible  and  the  IR- 
channel,  images  c)  and  d)  show  a  radar-out-the- 
window-view  (C-scope)  and  a  radar-map-view  (PPI- 
scope)  calculated  from  the  raw  radar  data,  which  are 
available  in  a  range  angle  (B-scope)  format. 

A  very  interesting  example  of  the  MMW-radar  capa¬ 
bility  for  obstacle  detection  is  presented  in  Fig.  4.4 
This  image  shows  the  same  taxi-way  on  Braunschweig 
airport  as  Fig.  4.3  c),  but  this  time  a  certain  obstacle 
was  located  in  the  region  of  the  taxi-way.  Fig.  4.4  also 
emphasizes  some  problems  which  are  connected  with 
the  interpretation  of  a  radar  image:  the  poor  image 
resolution  and  the  kind  of  image  generation  (range 
angle)  which  contains  no  information  about  the  ob¬ 
ject's  height  or  its  vertical  position.  The  only  informa¬ 
tion  which  is  really  transmitted  by  the  radar  image  of 
Fig.  4.4  is  a  reflex  in  a  part  of  the  scene  (taxi  way) 
where  no  reflex  would  be  expected. 

The  pilot  has  no  idea  about  the  kind  of  object  and 
due  to  the  way  of  image  generation  he  has  also  no 
information  about  the  object's  height  and  the  vertical 
position.  In  this  special  case  the  pilot  would  identify  a 
bird  sitting  on  the  taxi-way,  simply  by  looking  out  of 
the  window.  That's  of  course  not  the  idea  for  an  EVS- 
sensor,  but  from  the  pure  radar  information  nobody 
would  really  be  able  to  decide  whether  the  "obstacle" 
is  under,  ahead  of  or  above  the  own  position.  This 
problem  of  radar-image  interpretation  is  still  open. 
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Fig.  4.6;  Design  example  for  a  fused  EVS-display:  perspec¬ 
tive  radar  view  with  overlaid  terrain  data  and  PFD-sym- 
bology. 


4.3  Data  Fusion 

An  integrated  Enhanced  Vision  System,  as  it  is  defined 
in  chapter  2,  requires  the  fusion  of  several  data 
sources.  In  order  to  achieve  that,  all  data  has  to  be 
transformed  into  the  same  level  of  representation. 

Especially  a  radar  image  data  can  exist  in  different 
formats: 

1.  range-angle  B-scope  Fig.  4.5  a) 

2.  map  PPI-scope  Fig.  4.5  b) 

3.  out-the-window-view  C-scope  Fig.  4.5  c) 

Of  special  interest  for  radar  images  is  the  range-  an¬ 
gle,  or  B-scope  representation.  This  type  of  image 
contains  two  basic  information:  the  distance  (range) 
of  a  certain  object  relative  to  the  radar  head  and  the 
direction  in  terms  of  the  azimuth  angle.  These  "range- 
angle  "-images  use  the  x-axis  for  the  angle  representa¬ 
tion  and  the  y-axis  for  the  object  (i.e  reflector)  dis¬ 
tances. 

Figure  4.5.  shows  some  examples  for  these  different 
types  of  image  data  representations. 

A  first,  rather  simple  realization  of  an  Enhanced  Vision 
System  deals  with  the  fusion  of  radar  images,  which 
might  be  available  in  the  "PPI-scope "-format  or  as 
"out-the-window-view"  with  a  terrain  data  base. 

Position  and  attitude  delivered  by  the  vehicle's  naviga¬ 
tion  system  and  a  3-D  terrain  model  are  used  to  com¬ 


pute  a  wire  frame  overlay  of  terrain  elements  which 
can  be  presented  in  combination  with  the  radar  im¬ 
age.  The  fused  radar  images  are  a  rather  natural  sup¬ 
plement  to  the  type  of  information  which  are  carried 
by  the  primary  flight  display  (PFD).  A  tentative  exam¬ 
ple  how  to  integrate  such  a  fused  image  into  a  PFD  is 
shown  in  Fig.  4.6. 

Fusion  techniques,  especially  in  the  field  of  image  data 
fusion,  are  often  devoted  to  the  fusion  of  several  im¬ 
age  sources  with  different  spectral  sensitivity  [16],  In  a 
first  step  the  emission  of  all  objects  and  radiation 
sources  are  transformed  into  the  same  spectral  region 
in  order  to  make  them  visible  or  detectable  with  a 
common  tool,  which  is  in  many  cases  the  human  eye. 

Disregarding  the  possibility  of  data  reduction  which 
might  be  achieved  with  an  image  fusion  technique, 
the  main  reason  for  image  data  fusion  is  to  improve 
the  situational  awareness  of  the  pilot  and  the  observer 
respectively.  In  [18]  the  authors  demonstrate  a  dra¬ 
matic  increase  in  reliability  of  target  detection  and 
observer  performance  if  visible  and  thermal  images 
are  fused  either  grey  or  color. 

If  we  concentrate  for  a  moment  on  the  EVS-technique 
as  a  possibility  to  increase  the  pilot's  or  crew's  situa¬ 
tion  awareness  -  again  disregarding  requirements  for 
reconnaissance  or  remote  control  -  we  have  to  com¬ 
pete  with  a  very  famous  "vision  system":  the  pilot's 
eyes!  In  terms  of  situation  awareness  the  EVS- 
technology  should  "enhance"  the  ability  of  the  hu¬ 
man  operator,  but  not  replace  him.  With  this  back¬ 
ground  the  design  of  an  enhanced  vision  system, 
especially  the  applied  fusion  philosophy,  has  to  be 
examined  very  carefully : 


Fig.  4.7  :  Fusion  of  radar  images  to  an  overall  view  of 
Braunschweig  airfield  (fusion  in  space,  with  a  moving  vehicle). 
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Fig  4.8:  Noise  reduction  in  a  PPI-scope  image  (grid-size  20  m).  a)  detail  of  single  radar  image,  b)  the  same  part  after  fusion  of 
50  single  images  (approx.  3  seconds  of  data). 


•  What  kind  of  additional  sensor  really  enhances  the 
human  cognition? 

•  How  useful  is  a  sensor  which  covers  the  same,  or  a 
similar  spectral  range  as  the  human  eye? 

•  What  kind  of  representation  (map  or  "out-the- 
window-view")  in  what  situation  should  be  used? 

•  What  type  of  images  ,  symbols,  synthetic-  or  sen¬ 
sor  vision,  should  be  displayed  on  a  HUD? 

Another  rather  important  type  of  data  fusion,  espe¬ 
cially  for  radar  images,  is  related  to  a  technique  which 
could  be  called  "data  fusion  in  time  and  space".  Be¬ 
side  the  fusion  of  different  data  (images)  from  differ¬ 
ent  sources,  it  is  also  possible  to  fuse  different  images 
from  one  sensor,  taken  from  different  positions  and  at 
different  times  .  The  SAR-technique  or  computer  to¬ 
mography  are  good  examples  how  to  improve  the 
information  of  one  sensor  using  data  "fusion  in 
space".  For  EVS  applications  this  approach  offers  the 


reference 

system 

type  of  image 

Result 

cockpit 

C -scope 

noise  reduced  "out- 
the-window-view" 

geographic 

system 

PPI-scope 

overall  map-view 

Table  4.1:  Fusion  in  time  and  space  for  different  reference 
systems. 


following  advantages: 

•  (determination  of  objects  height  and  vertical  posi¬ 
tion  by  displacement  vectors), 

•  reduction  of  background  noise, 

•  integration  of  single  snapshots  to  an  overall  view. 

The  first  point  is  set  into  brackets,  because  the 
authors  feel  that  this  advantage  doesn't  really  exist, 
although  it  is  mentioned  in  the  literature  [10],  Because 
this  type  of  radar  measures  only  the  distance  to  cer¬ 
tain  objects,  an  information  about  their  vertical  posi¬ 
tion  (and  height)  is  not  available.  A  possible  solution 
of  this  problem  could  be  a  series  of  distance  meas¬ 
urements  which  are  taken  from  the  moving  radar, 
because  the  slant  distance  is  a  function  of  the  objects 
vertical  position  relative  to  the  radar  head.  This  is  the 
idea  to  overcome  the  lack  information  about  the  ver¬ 
tical  axis,  but  for  the  actual  available  radar  systems 
this  procedure  fails,  due  to  the  rather  large  range 
errors,  which  are  in  the  order  3-6m  [12], 

The  other  above  mentioned  data  fusion  techniques, 
fusion  in  time  and  fusion  in  space,  are  rather  promis¬ 
ing,  particularly  for  the  range-angle  images  of  a  radar. 
The  wire  frame  map  in  Fig.  4.7  represents  the 
Braunschweig  airfield  and  the  shaded  segments  indi¬ 
cate  those  parts  of  the  map  which  are  covered  by  the 
radar  (PPI-scope)  during  taxiing.  Using  the  information 
about  the  vehicles  position,  which  is  given  by  an  iner¬ 
tial  reference  unit,  or  by  means  of  GPS,  every  single 
radar  image  can  be  assembled  to  an  overall  view  of 
the  whole  airfield.  (For  this  special  fusion  technique 
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the  "map-like"  information  of  a  radar  image  is  very 
advantageous.) 

The  image  rate  of  the  DASA  HiVision  radar  is  16Hz, 
which  leads  to  a  large  overlap  of  the  PPI-scope  im¬ 
ages.  If  the  vehicle's  velocity  is  rather  slow  compared 
with  the  frame  rate,  which  is  a  typical  situation  during 
taxiing,  the  overlap  between  consecutive  radar  images 
covers  a  large  common  area.  This  feature  can  be  used 
for  noise  reduction  and  to  eliminate  or  to  detect 
moving  targets  by  means  of  an  averaging  technique, 
adding  up  weighted  grey  scale  values.  The  effect  of 
such  a  motion  compensated  fusion  technique  is  three¬ 
fold: 

•  noise  reduction, 

•  enlargement  of  image  coverage, 

•  increase  of  spatial  resolution. 

The  result  is  shown  in  Fig.  4.8;  a)  presents  a  detail  of  a 
raw  PPI-scope  image.  After  fusion  in  space  and  time, 
the  image  noise  was  nearly  completely  eliminated  and 
the  object  itself  can  be  identified  as  a  row  of  lamps. 

The  fusion  of  radar  images  shown  in  Figure  4.7  refers 
to  a  situation  with  a  fixed  map  and  moving  vehicle.  In 
this  case  the  fusion  procedure  eliminates  all  moving 
objects  and  of  course  the  noise.  These  features  quali¬ 
fies  this  type  of  fusion  as  a  "map  generator". 

The  same  fusion  technique  can  be  applied  to  a  "out- 
the-window-view"  (C-scope)  with  a  fixed  vehicle  and 
a  moving  map.  Unfortunately  moving  objects  are  also 
suppressed,  but  the  visibility  of  fixed  objects  will  in¬ 
crease.  Table  4.1  compares  these  different  fusion 
techniques. 


5.  Conclusion 

The  integrated  enhanced  vision  concept  as  it  is  inves¬ 
tigated  at  DLR  demands  the  fusion  of  sensor  vision, 
synthetic  vision  and  additional  information  of  the 
inner  and  outer  status  of  the  vehicle.  In  order  to  meet 
most  weather  situations  DLR  decided  to  use  the  HiVi¬ 
sion  radar  from  DASA  Ulm,  which  is  a  fon/vard  looking 
active,  but  from  the  military  viewpoint  a  "silent", 
MMW  imaging  sensor.  Radar  sensors  are  very  prom¬ 
ising  in  the  field  of  weather  penetration,  but  on  the 
other  side  they  are  a  great  challenge  concerning  noise 
and  resolution,  (angle  and  range)  and  they  fail  com¬ 
pletely  delivering  the  vertical  position  of  the  depicted 
object.  Data  fusion  techniques,  such  as  an  overlay 
with  terrain  data,  or  fusion  in  time  and  space  may  be 
applied  to  overcome  the  former  difficulties.  The  re¬ 
striction  of  the  MMW-technology  to  a  range-angle 
information  and  the  impact  on  the  architecture  of  the 
overall  Enhanced  Vision  System  is  still  a  topic  for  re¬ 
search. 

A' very  promising  imaging  technique  in  this  fieid 
would  be  the  total  3-D  reconstruction  of  the  outside 
world  from  2-D  (range-angle)  radar  images.  This 


would  require  the  adoption  of  signal  reconstruction 
methods,  as  they  are  known  from  computer  tomog¬ 
raphy  (CT)  by  fusing  radar  images  with  different  scan¬ 
ning  directions.  These  image  data  could  be  acquired 
for  example  by  at  least  two  orthogonal  mounted  ra¬ 
dar  antennas  or  by  a  rotating  one. 

First  experiments  with  the  DASA  HiVision  Radar  as  an 
enhanced  vision  sensor  show  some  very  promising 
results  concerning  range,  image  rate  and  resolution. 
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1.  SUMMARY 

processing  and  time  requirements  of  computer  systems 
increase  over  borders  of  single  processor  architectures,  it 
is  becoming  more  attractive  to  use  distributed  computing 
with  additional  real-time  capabilities.  In  several  cases, 
traditional  programming  languages  have  become 
insufficient  to  build  such  systems  easily,  especially  when 
basic  software  quality  factors  such  as  reliability, 
correctness,  robustness,  ease  of  design,  development, 
testing  and  maintenance  are  concerned.  In  this  paper 
basic  issues  relevant  to  distributed  systems  are  discussed, 
similar  systems  and  languages  are  compared,  a  new 
language  with  its  supportive  run-time  system  is 
introduced.  The  new  system  provides  a  solution  that  hides 
implementation  details  from  the  programmer  by 
embedding  distribution  and  real-time  issues  within  the 
programming  language  structure.  In  order  to 
demonstrate  the  usage  of  the  new  language,  basic 
requirements  of  distributed  naval  command  and  control 
systems  are  touched  and  a  simple  naval  application 
design  and  part  of  the  implementation  is  briefly  explained. 

2.  INTRODUCTION 

Real-world  systems  consist  of  many  distributed 
components  that  are  asynchronously  interacting  with 
their  environment.  When  distributed  computing  and 
concurrent  processing  are  considered,  it  can  be  seen 
that  conventional  programming  languages  have 
become  insufficient  due  to  the  lack  of  convenient 
structures  for  inter-process  and  inter-node 
communication.  These  languages  may  be  efficient 
for  software  architectures  that  use  single  processor. 
However,  when  it  is  necessary  to  distribute  an 
application  over  a  network,  they  tend  to  become  a 
burden  on  the  programmers  due  to  inconvenient 
facilities  of  the  underlying  operating  system  and 
hardware.  Therefore  it  becomes  essential  to  use  a 
middleware  for  decoupling. 

Generally,  applications  are  developed  using  a 
classical  language  and  several  user  defined  libraries. 
In  this  case,  the  importance  of  software 
characteristics,  methodology  and  development 
issues  get  the  focus.  One  of  the  most  suitable 


programming  styles  is  to  use  object-orientation.  The 
distributed  form  of  this  approach  requires 
communication  between  distinct  objects  on  different 
nodes.  Tightly  or  loosely  coupled  computer  systems 
may  be  used  depending  on  the  inter-object 
communication  load.  Real-time  applications  that 
require  critical  input  and  output  operations  and 
processing  on  different  nodes  are  especially  difficult 
to  design  and  implement.  Such  systems  are  also 
required  to  have  important  features  like  high 
reliability,  efficiency,  scalability,  expansibility, 
portability  and  ease  of  maintenance.  Therefore, 
distributed  programming  languages  have  evolved  to 
meet  the  new  requirements  within  the  language 
constructs,  leaving  only  the  design  part  to  the 
programmer.  Since  object-oriented  programming 
(OOP)  has  several  important  capabilities,  adapting 
them  onto  a  distributed  environment  with  real-time 
properties  has  become  an  important  area  of  research, 
which  has  resulted  into  the  evolution  of  a  number  of 
object-oriented,  concurrent,  distributed  and  real¬ 
time  programming  languages. 

One  important  domain  for  research  is  the  command 
and  control  systems  (C^)  that  facilitates  a  real-time 
middleware.  The  system  introduced  in  this  paper 
proposes  a  solution  without  an  explicit  middleware 
approach,  but  a  programming  language. 

3.  DISTRIBUTED  AND  REAL-TIME  COMPUTING 
A  distributed  and  real-time  system  is  desired  to  be 
concurrent,  fault-tolerant,  reliable,  fast  and  efficient, 
in  both  computing  and  communication. 
Programming  languages  that  are  used  to  develop 
such  systems  should  have  some  extra  features,  in 
addition  to  software  engineering  properties,  like 
modularity,  extensibility  and  reusability.  Some  of 
these  features  are  discussed  below. 

3.1  OOP  Languages 

OOP  languages  have  special  constructs  which  divide 
programs  into  smaller  portions  called  objects.  These 
objects  can  only  be  accessed  via  methods  that  are 
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defined  in  their  interface.  Common  properties  of 
OOP  languages  are  specified  in  (Meyer  1988).  A 
program  must  be  built  using  separate  modules  that 
help  designers  to  produce  software  systems  made  of 
autonomous  elements  connected  by  a  coherent 
simple  structure  (Modularity).  Objects  are  defined  as 
abstract  data  types  encapsulating  data  elements  and 
operations  on  them  (Data  abstraction).  Objects  using 
system  resources  are  deallocated  automatically 
when  they  are  finished  with  (Memory  management). 
Nonsimple  types  whose  instances  are  objects  define 
classes  (Class).  A  class  may  be  derived  from  another 
class  inheriting  some  or  all  of  the  properties  of  the 
base  class  (Inheritance).  Finally,  polymorphism 
refers  to  the  ability  of  an  entity  to  refer,  at  mn-time, 
to  instances  of  various  classes.  Pure  OOP  languages 
meet  all  these  criteria,  providing  multiple  and 
repeated  inheritance.  Having  the  first  four  properties 
and  all  properties  are  called  object-based  and  object- 
oriented  respectively.  When  these  properties  are 
applied  to  distributed  environments,  it  becomes 
essential  to  define  objects  either  as  activities  or  data. 
Many  concurrent  languages  like  RTC-H-t-  (Ishikawa 
and  Tokuda  1992),  Mentat  (Grimshaw  1993), 
Concurrent  Smalltalk  (Yokote  1990)  and  Eiffel  have 
chosen  to  use  objects  for  processing  elements  that 
are  subjected  to  distribution.  On  the  other  hand, 
Linda  (Robinson  and  Arthur  1995)  and  SR 
(Andrews  1982)  use  data  objects. 

3.2  Concurrency 

Concurrent  languages  use  special  constructs  for 
creating  processes  depending  on  the  underlying 
operating  system.  A  process  is  usually  defined  as 
Unix-like  processing  elements  that  have  separate 
contexts  with  distinct  address  spaces  coexisting  on 
the  same  processor.  Some  systems  allow  threads  to 
be  used  within  a  process  to  support  higher  level  of 
concurrency.  Mutual  exclusion  and  synchronization 
must  be  provided  when  concurrent  accesses  to 
critical  resources  are  considered.  Although  operating 
systems  can  provide  such  facilities,  it  is  usually  the 
programmer’s  responsibility  to  implement  shared 
data  protection  and  process  synchronization.  In 
distributed  systems,  processes  communicate  by 
message  passing  or  remote  procedure  calls. 
Maximum  parallelism  can  be  achieved  by  providing 
asynchronous  message  communication  that  lets  the 
sender  continue  without  having  to  wait  for  reply. 
Some  concurrent  languages  let  the  programmer 
specify  the  concurrency  with  the  language 
constructs,  while  others  manage  it  implicitly. 
Processes  and  data  storage  areas  can  be  distributed 
over  the  network.  These  objects  can  be  moved,  or 
migrated,  from  one  node  to  another. 


3.3  Process  management 

When  parallel  computing  is  considered,  process 
management  requires  special  attention  as  it  is 
necessary  to  manage  them  different  than  ordinary 
objects.  For  example,  in  C-f+,  objects  are  created  by 
their  constructors  and  deleted  by  their  destructors. 
Calling  a  class  constructor  that  creates  a  process  and 
a  destructor  that  kills  the  process  is  not  a  suitable 
solution  in  concurrent  environment  due  to  extensive 
overhead  of  process  management.  Therefore,  some 
concurrent  programming  languages  create  or 
terminate  processes  explicitly,  while  others  create 
them  implicitly,  but  terminate  explicitly.  Process 
granularity  is  another  design  parameter  in  concurrent 
programming.  If  communication  to  processing  ratio 
of  a  program  is  small  then  coarse  or  large  granularity 
must  be  selected.  If  this  ratio  is  high,  then  small  or 
fine  granularity  can  be  used.  Threads  may  also  be 
used  to  obtain  finer  granularity.  Coarse  granularity 
typically  implies  one  task  per  object,  requiring  less 
communication. 

3.4  Communication 

In  a  concurrent  environment,  objects  that  are 
implemented  as  processes  use  interprocess 

communication  facilities  of  the  underlying  operating 
system.  Concurrent  languages  like  Mentat,  ES-Kit 
C-i-P  (Smith  and  Chatterjee  1990)  and  Pearl 

(Stoyenko  and  Halang  1993)  have  constructs  that 
provide  this  communication  in  synchronous  or 
asynchronous  manner.  Ada,  which  is  not  a 

distributed  language,  uses  rendezvous  mechanism, 
like  SR,  to  exchange  information  between  tasks.  In  a 
distributed  environment,  objects  communicate  using 
messages  that  are  passed  through  various  network 
layers.  Distributed  operating  systems  hide  the 

network  level  communication  so  that  all  programs 
seem  to  execute  on  a  single  machine. 

When  objects  are  implemented  as  distinct  processes 
or  shared  memory  segments,  they  have  to  be 
identified  specifically  at  the  operating  system  level, 
even  at  the  network  level.  Therefore  a  global  name 
resolution  with  kernel  or  run-time  system  support  for 
message  communication  has  to  be  used.  Some 
systems  use  direct  communication  method,  in  which 
messages  are  sent  to  the  receiver  whose  name  and 
location  is  known  by  the  sender.  This  approach 
increases  object  dependency  as  each  object  must 
have  sufficient  and  consistent  information  about 
other  objects. 

3.5  Inheritance 

Many  concurrent,  object-oriented  languages  have 
problems  with  inheritance.  Some  languages  such  as 
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Eiffel,  ES-Kit  C-i-i-,  Java,  though,  eliminate  multiple 
inheritance  where  some  languages,  like  Orca, 
POOL-T  (America  1987)  do  not  allow  even  single 
inheritance.  Other  approaches  like  delegation,  static, 
dynamic  and  on-demand  inheritance  and  recipe- 
query  method  are  also  used  by  some  languages. 

3.6  Real-time  aspects 

Some  mission  critical  systems  require  real-time 
constraints  to  provide  fast  response  to  events 
occurring  at  non-regular  rates.  Real-time  systems 
can  be  separated  into  three  groups:  Soft  real-time 
systems  do  not  fail  in  case  response  time  constraint 
is  not  met,  only  performance  is  degraded.  Hard  real¬ 
time  systems  have  to  meet  deadlines,  otherwise 
system  failure  occurs.  Firm  real-time  systems  are 
somewhere  in  between,  where  low  probability  of 
missing  a  deadline  can  be  tolerated.  Generally,  hard 
real-time  systems  are  designed  to  work  with  special 
hardware  because  of  the  importance  of  deadlines. 
Timing  constraints  for  such  systems  are  predicted 
off-line  and  sufficient  resource  is  always  left  for 
unpredictable  events.  Therefore  resource  allocation, 
pre-emption  and  priority  issues  are  very  important  to 
meet  the  real-time  requirements. 

Languages  used  for  building  real-time  systems  are 
desired  to  have  some  important  characteristics  such 
as  strong  typing,  dynamic  memory  management,  fast 
parameter  passing  techniques,  exception  handling, 
abstract  data  types  and  modularity. 

4.  RELATED  WORK 

A  number  of  concurrent,  distributed  and  real-time 
systems  as  well  as  programming  languages  have 
been  examined.  Of  these,  RTC-h-i-  and  Mentat  are 
emphasized  as  they  have  more  similarities  with  the 
proposed  approach.  Several  common  points  are 
noted  and  the  language  structure  with  its  run-time 
communication  facilities  are  considered. 

RTC++  introduces  temporally  constrained  active 
objects  which  are  distinguished  from  ordinary  C-i-i- 
objects.  Timing  constraints  may  be  specified  as  part 
of  method  declarations.  The  language  also  allows 
timing  constraints  to  be  specified  in  commands 
within  methods.  RTC-I-+  uses  a  multi-threaded 
model  to  describe  real-time  applications  which 
means  multiple  operations  can  be  invoked  on  an 
object  and  alter  its  state.  Asynchronous 
communication  is  not  supported  due  to  the  difficulty 
of  recovery.  The  language  employs  priority 
inheritance  distribution  facilities. 

Mentat  is  an  object-oriented,  parallel  processing 
system  using  MPL,  an  OOP  language  based  on  C-i-t-, 


masking  the  complexity  of  the  parallel  environment 
from  the  programmer.  The  underlying  Mentat  ran- 
time  system  provides  a  virtual  machine  abstraction 
for  easy  portability  to  new  architectures.  Mentat 
extends  C-)-+  in  three  ways:  the  specification  of 
Mentat  classes,  the  rtf{)  value  return  mechanism,  and 
the  select/accept  statement.  The  programmer  uses 
Mentat  objects  just  as  any  other  C-t-f  object. 
Independent  Mentat  objects  have  a  distinct  address 
space,  a  system  wide  unique  name,  and  a  thread  of 
control. 

Java  from  Sun  Microsystems  is  a  pure  object- 
oriented  language  specially  designed  to  work  on 
heterogeneous  internet  nodes  executing  byte  code.  A 
large  class  library  is  provided  for  several  purposes. 
Concurrency,  portability,  polymorphism  and 
dynamic  linking  is  emphasized,  but  due  to  byte  code 
interpretation,  real-time  efficiency  is  reduced. 

CORBA  (Common  Object  Request  Broker 
Architecture)  is  a  widely  excepted  architecture 
standard  utilizing  a  special  Interface  Definition 
Language  (IDL).  Even  though  CORBA  uses  IDL  for 
interfaces,  programmers  have  to  use  a  common 
language  like  C-t-f-  or  Java  for  implementation. 

5.  CORD  PROGRAMMING  LANGUAGE:  CPL 

The  available  research  work  has  been  examined  and 
it  is  concluded  to  combine  several  features  into  one 
programming  language  as  an  extension  to  C-i-i-  with 
new  keywords.  Since  a  higher  level  of  abstraction  to 
logically  interconnect  distributed  processors 
considering  real-time  requirements  is  needed,  a 
special  layer  running  on  top  of  the  operating  system 
has  been  built.  This  layer  is  called  the  CORD-RTS 
standing  for  Concurrent,  Object-oriented  and  Real¬ 
time  Distribution  Run-Time  System.  Therefore  the 
language  is  given  the  name  CPL  (The  CORD 
Programming  Language). 

5.1  The  CORD  System 

The  CORD  System  framework  is  designed  to  allow 
application  development  that  is  independent  of 
hardware,  operating  system  and  network  topology, 
facilitating  a  programming  language. 


Figure  1.  The  CORD  system  architecture 
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The  system  has  less  interaction  with  the  underlying 
operating  system.  Therefore,  the  entire  system, 
together  with  applications,  can  easily  be  ported  to  a 
new  environment  by  modifying  only  those  parts  that 
are  in  interaction  with  the  operating  system.  Figure  1 
shows  the  system  architecture  which  consists  of  the 
following  parts: 

The  CORD  Run-Time  System  (RTS)  :  The  RTS 
running  on  each  node  has  four  active  elements: 
Object  Manager  is  the  general  object  controller 
which  maintains  a  distributed  database  of  the 
running  programs,  classes  and  their  instances.  It  also 
manages  the  nodal  data  storages  in  shared  memory, 
handles  prioritized  messages,  controls  the  publish- 
subscribe  mechanism  and  reports  node  availability. 
Net  Manager  handles  network  access  by  considering 
the  logical  CORD  network.  Device  Manager,  which 
is  executed  optionally,  keeps  track  of  devices  on  the 
node  and  provides  multiple  access  to  them.  Error 
Manager  collects  reported  errors  in  a  specific  format 
and  displays  or  writes  into  a  disk  file.  An  interactive 
command  shell  is  also  provided  to  execute  programs 
and  objects  individually  in  addition  to  obtaining 
system  information.  These  elements  with  their 
interactions  are  shown  in  Figure  2. 


Figure  2.  The  CORD  system  infrastructure 


Main  program  :  A  program  is  defined  as  a  collection 
of  executables  which  will  later  construct  the  global, 
distributed  program.  Therefore,  each  CPL  program 
must  have  a  main  module,  like  a  conventional 
program,  which  ultimately  is  executed  on  the  RTS. 
This  part  registers  the  program  itself  and  its  classes 
to  the  RTS,  and  then  creates  the  necessary  objects. 
Those  objects  later  create  other  objects  over  the 
network.  Terminating  the  main  program  results  in 
the  temiination  of  all  its  classes  and  objects, 
independent  of  their  location. 


Class  Servers  :  In  order  to  create  objects  as 
processing  units,  a  server  is  used  from  which  child 
processes  are  forked.  There  are  as  many  active  class 
servers  as  the  number  of  active  classes  declared  in  a 
program.  Each  node  has  a  copy  of  each  server,  ready 
to  create  an  object. 

Objects  :  There  are  three  basic  types  of  objects  in 
CPL.  Active  objects  are  the  processing  elements, 
passive  objects  are  plain,  shared  data  storage  areas. 
Device  objects  control  the  hardware  part  of  the  node 
and  are  kept  inside  the  RTS  of  a  node. 

5.2  Classes 

CPL  introduces  three  new  types  of  classes  in 
addition  to  regular  C-H-  classes.  These  are  active, 
passive  and  device  classes,  from  which  active, 
passive  and  device  objects  are  created  respectively. 
CPL  also  supports  multiple  inheritance  in  a  way 
similar  to  C-i-i-,  provided  that  the  classes  are  of  the 
same  type.  Inheritance  is  achieved  statically  by  the 
CPL  compiler. 

Active  class  :  Active  classes  are  implemented  using 
Unix-like  processes  and  act  as  primary  processing 
elements.  Instances  of  this  type  of  classes  are 
capable  of  calling  methods  of  other  objects.  Methods 
of  these  objects  are  accessed  by  only  special 
messages  generated  by  the  compiler,  however  the 
programmer  just  issues  a  method  call.  Instances  of 
active  classes  may  have  static  or  dynamic  data  parts 
that  are  specified  in  the  class  definition.  Dynamic 
data  are  stored  in  a  reliable  storage  on  the  node. 
Static  data  elements  are  stored  inside  the  object 
context.  During  object  migration,  the  dynamic  data 
part  is  copied  to  a  new  location  in  the  destination 
node  and  a  new  active  object  is  created  initializing 
from  the  data  copied  to  that  location.  Active  objects 
can  be  accessed  from  any  node  of  a  distributed 
environment  provided  that  the  necessary  naming  and 
scope  resolution  is  achieved. 

Passive  class  :  Instances  of  passive  classes  are  used 
as  a  means  of  data  storage  and  are  not  capable  of 
calling  methods  of  other  objects.  These  objects  can 
easily  be  accessed  within  the  context  of  an  active 
object,  providing  fast  read  or  write  functions. 
Passive  objects  can  only  be  accessed  by  an  active 
object  located  on  the  same  node  and  cannot  be 
migrated. 

Device  class  :  A  device  class  defines  a  coherent 
interface  for  input  and  output  devices.  The  class 
definition  specifies  only  the  necessary  parts  for  a 
programmer  to  access  a  physical  device.  All  controls 
are  done  by  the  Device  Manager  existing  on  a  node. 
The  manager  can  also  provide  active  object 
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subscription  mechanism  for  read  access.  An  active 
object  that  wants  to  be  triggered  by  a  device  input 
subscribes  for  that  device  object  data  with  its  own 
method.  When  new  data  is  read,  the  manager  calls 
the  registered  method  of  the  active  object. 

5.3  Communication 

Inter-object  communication  can  be  synchronous, 
asynchronous  or  broadcast.  These  modes  are  set  by 
the  CPL  compiler.  Actually,  programmers  never  use 
messages,  instead,  method  calls  specified  in  the  class 
definitions  are  used.  All  method  calls,  replies  and 
published  methods  are  converted  into  system  level 
messages  having  priority,  destination  and  time 
stamp.  Therefore  all  method  parameters  have  an 
indicator,  in,  out  or  inout,  to  specify  the  type  of 
communication.  If  a  method  has  only  in  parameters 
or  no  parameter,  it  causes  a  one-way  call.  If  there  is 
at  least  one  out  or  one  inout  parameter,  then  the  call 
is  said  to  be  two-way  and  blocked.  Subscription  is  a 
way  of  asynchronous  method  invocation  without 
having  to  know  the  destination  objects. 

Interdependency  between  objects  is  reduced  by 
allowing  communication  only  between  active 
objects  and  the  RTS.  Thus,  an  object  prepares  a 
message  for  a  specific  method  call  indicating  the 
destination  and  sends  it  to  the  RTS.  The  sender 
object  does  not  have  to  know  where  the  receiver 
object  is  located,  or  in  what  state  it  is.  The  RTS  finds 
the  location  of  the  destination  object  and  forwards 
the  message  either  to  the  object  itself  (if  it  is  located 
on  that  node)  or  to  the  RTS  of  the  node  on  which  the 
object  resides.  The  RTS  on  the  destination  node 
transfers  the  message  to  the  object.  The  destination 
object  may  be  in  a  blocked  state  waiting  for  a  method 
invocation  or  a  reply  to  a  specific  call.  It  can  also  be 
in  a  running  state  doing  its  work.  In  any  case,  the 
incoming  call  is  accepted  implicitly  by  the  object, 
and  the  message  is  put  into  a  queue  inside  the  object 
context,  waiting  to  be  processed  according  to  its 
priority.  No  preemption  takes  place  inside  the  object. 

5.4  Real-time  aspects 

CPL  and  its  supportive  run-time  system  is  designed 
to  develop  soft  real-time  systems  easily.  Therefore, 
some  basic  timing  constructs  are  defined  in  CPL  for 
specifying  real-time  constraints  at  statement  level. 
Prioritized  message  handling,  timed  loops,  time- 
triggered  method  calls  in  millisecond  domain  are 
provided  and  process  scheduling  is  left  to  the 
existing  operating  system.  Language  structures  also 
enable  the  programmer  to  specify  actions  in  case 
time  limit  for  a  statement  or  for  a  method  call  are 
exceeded.  Fast  access  to  shared  data  is  achieved 
using  passive  objects. 


All  active  objects  send  their  messages  carrying 
method  calls  to  the  RTS  with  an  assigned  priority 
ranging  from  1  to  10  and  a  time-out  value  defining 
the  validity  of  the  message.  The  programmer  should 
assign  suitable  priorities  and  time-out  values  to 
method  calls  explicitly  for  efficient  resource 
utilization,  otherwise,  system  defaults  are  used. 
Unless  otherwise  is  stated,  active  objects  send  their 
messages  with  their  own  scheduling  priority.  Since 
messages  are  subjected  to  communication  delays,  in 
some  cases,  programmers  can  assign  priorities  to 
method  calls  higher  than  the  active  object’s. 

Priority  inversion  feature  is  provided  for  the  objects 
that  are  running  with  lower  priority.  When  a  high 
priority  call  is  received,  the  receiving  object 
increases  its  scheduling  priority  up  to  the  priority  of 
the  incoming  call.  After  the  method  is  executed,  the 
priority  is  lowered  again.  Low  priority  calls  do  not 
cause  an  object  to  reduce  its  scheduling  priority. 

Real-time  filters  are  used  inside  the  CORD  system. 
Each  manager  in  the  RTS  checks  the  time-out  value 
of  an  incoming  message  to  decide  if  there  is 
sufficient  time  for  communication  delay  and 
processing.  If  not,  it  does  not  forward  the  message 
and  sends  a  system  reply  back  to  the  sender.  The 
sender  object  then  raises  an  exception  to  be  handled 
by  the  programmer. 

5.5  Distribution 

The  CPL  objects  can  be  created  on  any  node  within 
the  network.  Object  creation  is  performed  in  a 
similar  way  used  in  C-i-t-  constmctors.  Objects  use 
three  level  naming  convention.  Of  these,  the  first 
level  uses  program  identification  which  enables 
multiple  copies  of  the  same  distributed  program  run 
simultaneously.  The  second  level  is  the  class 
identification,  and  the  third  level  is  the  object 
identification.  In  order  to  speed  up  processing, 
numeric  representation  of  logical  names  are  used 
within  the  system  transparently. 

Migration  of  active  objects  is  performed  by  stopping 
the  process,  copying  the  contents  of  the  dynamic 
data  and  starting  a  new  process  at  the  new  location. 
For  this  purpose  active  classes  specify  if  they  want  to 
use  dynamic  data  storage  which  is  in  a  safe  memory 
area.  Necessary  communication  and  control  of 
messages  during  migration  are  handled  by  the  RTS. 

CORD  objects  do  not  communicate  directly  with 
each  other.  The  caller  object  only  issues  a  method 
call  with  appropriate  parameters  and  waits  for  reply 
from  the  RTS.  If  fast  information  transfer  in  the 
same  node  is  needed,  passive  objects  can  be  used 
without  interacting  with  the  RTS. 
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5.6  System  Characteristics 

The  CPL  compiler  generates  C-i-i-  source  code 
modules  to  be  compiled  with  the  target  compiler. 
The  resultant  distributed  program  consists  of  a 
number  of  executables  for  the  main  module  and  each 
active  class.  These  program  components  are  loaded 
into  the  disk  locations  which  are  accessible  by  all  the 
nodes.  The  software  structure  of  an  application 
developed  in  CPL  is  shown  in  Figure  3.  Application 
is  isolated  from  the  operating  system,  underlying 
computer  hardware  and  local  area  network. 
However,  standard  C-r-t-  libraries  as  well  as  operating 
system  calls,  can  directly  be  used  in  addition  to  the 
CORD  Interface  in  C-i-f. 


— 

CPL  Program 

Stub  Code,  CORD 

(Class  server)  Interface 

C+-H 

Libraries 

Operating 

System 

Calls 

CORD-RTS 

Operating  System 

Computer  Hardware  /  LAN 

Figure  3.  Software  architecture 


Generally,  a  good  OOP  practice  dictates  an  efficient 
use  of  scope  resolution.  That  is,  objects  must  be 
visible  to  only  those  objects  that  are  in  the  same 
block.  An  active  object  is  a  process  managed  by  the 
operating  system  of  that  processor.  It  can  be 
accessed  by  any  other  active  object  within  the  same 
network.  This  seems  in  contradiction  with  some  of 
the  software  engineering  rules  like  abstraction  and 
information  hiding.  However,  CPL  classes  use  the 
same  visibility  rules  as  C-i-f.  Even  though  created 
objects  are  global  to  all  nodes  the  CPL  compiler 
manages  the  scope  rules  considering  files,  functions 
and  blocks. 

CPL  enables  programmers  to  be  able  to  develop  a 
distributed  program  without  concentrating  on 
communication  and  the  physical  layout  of  the 
system.  Programmers  write  CPL  modules  as  if 
writing  a  regular  C-I--I-  module,  either  using  full 
capabilities  of  C-n-  or  CPL  specific  keywords  and 
structures  for  distribution.  CPL  modules  are  then  fed 
to  the  CPL  compiler  to  generate  separate  client  and 
server  stub  code  to  access  the  RTS.  The  resultant 
C-i-f-  modules  are  compiled  with  an  existing  C+-1- 
compiler  to  generate  the  necessary  executables  for 
each  active  class  and  for  the  main  module.  Batch 
processing  hides  details  from  programmer.  These 
files  are  then  loaded  onto  the  target  nodes.  After 
starting  up  the  RTS  on  each  node,  the  main  program 
is  loaded  and  started  from  the  interactive  shell  which 


creates  class  servers  and  objects.  Sample  CPL  class 
declarations,  object  creation,  timed  loops  and 
statements  with  time  constraints  are  shown  below: 

active  class  Class_B  inherits  Class_A  { 
uses  Classic;  //include  the  interface 

priority  5;  //object  scheduling  priority 

static:  //context  specific  data  part 

int  men\_.size,  nuin_elem; 

Boolean_Type  ready; 

dynamic  1024:  //amount  of  dynamic  memory 

Link_List  element_list; 

methods:  //class  methods 

GetNumOfElem{out:  int  n) ;  //two-way  call 

Compute(in:  int  X;  out:  int  Y) ;  //two-way  call 
Calculate (inout:  int  Num) ;  //two-way  call 

Insert (in:  int  Num);  //one-way  call 

periodic  1000:  //time-trigger  in  millisecond 

Do_Processing ( ) ; 

pxihlish:  //published  data 

NewData(int) ;  //parameter  type  is  used 

subscribe:  //own  method  to  another  class 

Calculate  to  Class_C .New_Data;  //active  class 
Insert  to  Serial_Comm . TheDevice . Read_Data ;  / /device 
private: 

void  SquareO;  //visible  to  only  this  class 

}; 

passive  class  Pi  { 

static:  //passive  class  data 

int  index; 

int  data[100] ; 

methods: 

Insert(in:  int  Value); 

Get(in:  int  Key;  out:  int  Value); 

); 

device  class  Serial_Comm  { 

read  int;  //data  type  to  read  from  device 

buffer  10;  //number  of  elements  in  buffer 

speed  19600;  //baud  rate  (bps) 

methods:  //callable  method 

GetValue (out :  int  val); 
pxxblish: 

Send_Data(int) ;  //published  when  data  is  read 

}; 

Class_A  objl;  //New  creation 

Class_A  obj3 (initial_val)  on  "N0DE_4"; 

Class_A*  obj4  =  new  Class_A  on"NODE_4"; 

objl.BindC'objl'') ;  //Binding  to  an  existing  object 
objl. GetA(val, TIMEOUT) ;  //millisecond,  timed  call 
objl.GetA(val)  timeout  1000  {  HandleTimeout ( ) ;  } 
objl.SetA(val)  on  15:30:00-0; 
objl.Calculate(a,  b)  within  50; 

do  { 

Sensor .Check_Status (status)  timeout  100; 

}  every  1000;  //millisecond 

do  { 

a .  Read ( ) ; 

b. WriteO  ; 
sleep (1) ; 

)  until  20:45:00.0; 

In  addition  to  the  keywords,  some  basic  functions  for 
CPL  classes  and  RTS  utility  routines  are  also 
provided.  The  programming  process  will  be  less 
problematic  if  certain  functionalities  can  be  achieved 
through  the  use  of  keywords,  rather  than  calls  to 
library  functions.  Also,  learning  the  rules  of  a 
language  is  easier  than  learning  a  given  library. 
Since  CPL  is  based  on  C-t-i-  language,  any  previously 
written  C-t-H  code  is  accepted  by  the  CPL  compiler. 

6.  PERFORMANCE  ANALYSIS 

During  the  system  development,  various  tests  are 

performed  in  order  to  figure  out  the  system 
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bottlenecks  and  time  consuming  parts.  Table  1 
shows  some  performance  figures  in  microseconds 
obtained  by  averaging  the  time  consumed  by  one 
object  calling  another  object’s  method.  The  first  test 
environment  consists  of  Sun  Sparc4  (100  MHz) 
workstations  with  SunOS  4.1.3  on  10  Mbs  ethemet. 
The  second  one  is  a  stand-alone  Sparc  notebook  (70 
MHz)  with  Solaris  2.5  and  the  third  one  consists  of 
Sun  UltrabO  (300  MHz)  workstations  with  Solaris 
2.5  on  155  Mbs  ATM  network  with  LAN  emulation. 

The  performance  tests  have  shown  that  inter-object 
communication  can  be  reduced  dramatically  if 
passive  objects  are  used.  However,  active  object 
communication  latency  increases  as  the  number  of 
active  objects  requiring  parallel  execution  increases. 


Table  1.  Performance  figures 


Test 

Sparc4 

Sp.Book 

Ultra60 

One-way  method  call, 
same  node 

850 

1300 

550 

Two-way  method- 
call,  same  node 

2600 

3500 

1550 

Two-way  method- 
call,  different  nodes 

6700 

N/A 

2250 

Reading  passive  object 
with  Lock/Unlock 

180 

220  ‘ 

50 

Reading  passive  object 
without  Lock/Unlock 

0.18 

0.25 

0.15 

7.  COMMAND  AND  CONTROL  SYSTEMS 

Computerized  control  systems  provide  data 
collection  from  various  sensors  or  input  devices, 
evaluate  them  and  control  some  actuators. 
Command  and  control  (C^)  systems  in  military 
platforms  help  a  command  team  to  understand 
tactical  situations  and  allow  them  to  control  sensors, 
weapons  and  other  subsystems.  Recent  C^  systems 
include  communication  as  well.  Such  systems  have 
some  common  characteristics  when  technical  and 
functional  aspects  are  considered. 

These  systems  are  difficult  to  develop  because,  their 
use  in  defense  applications  makes  correctness  and 
reliability  critical.  Their  requirements  are  difficult  to 
determine  and  standardize  as  they  are  influenced  by 
organizations,  policies  or  doctrines.  Their  design 
require  special  techniques  to  guarantee  real-time 
constraints  are  met.  As  with  any  large  system, 
realizing  such  systems  require  high  costs  because  of 
the  amount  of  mission  critical  software  to  be 
developed.  Modem  systems  have  distributed 
processing,  real-time  constraints,  multiple  and 
predefined  hardware  interfaces  and  operational 
requirements.  A  system  for  naval  platforms  has 


several  external  interfaces  to  the  user  and  to  the 
platform  sensors,  weapon  systems,  navigation 
system  and  data  links  as  shown  in  Figure  4. 


Figure  4.  Context  diagram  of  a  C&C  system 


The  primary  object  for  a  system  used  for 
surveillance  is  a  track,  which  is  defined  as  the 
representation  of  an  external  object  in  the  system 
with  several  characteristics  like  type,  position, 
kinematics  and  identity.  The  system  maintains  a 
common  track  database  which  is  updated 
continuously  by  the  sensors.  This  database  is 
managed  by  a  special  software  component,  the  Track 
Management,  which  provides  correlation  of  multi¬ 
sensor  track  data  to  form  a  single  track  for  the  user. 
Tracks  are  identified  uniquely  by  assigning  numbers 
considering  their  types,  such  as  point  tracks,  bearing 
tracks,  reference  positions,  operator  maintained 
tracks  and  network  tracks. 

8.  PROPOSED  SOLUTION  TO  APPLICATIONS 

Large  scale  system  software  requires  efficient 
infrastructure  that  is  capable  of  handling  large 
amount  of  high  frequency  data  preserving  real-time 
aspects.  The  CORD  System  can  meet  requirements 
for  a  soft  real-time,  distributed  application  well 
enough  providing  necessary  decoupling  of 
hardware,  operating  system  and  application  software 
with  a  modular  approach.  CPL  brings  extra 
advantages  by  providing  a  straightforward  solution 
with  programming  language  abstraction  to  the 
complexity  of  design  and  implementation. 

A  common  naval  system  on  board  a  combat  ship 
receives  asynchronous  data  from  radars  and  other 
sensors,  processes  them  considering  priorities  and 
sends  control  commands  to  actuators,  such  as 
trackers,  weapons  or  other  peripherals. 

8.1  Architecture 

There  are  many  different  architectural  solutions  to 
naval  applications.  These  can  be  addressed  in  two 
separate  topics  as  hardware  and  software. 

Hardware  Architecture 

Recent  combat  systems  utilize  loosely  coupled 
hardware  structure.  Therefore  a  local  area  network  is 
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used.  Computation  elements  are  usually  placed  over 
the  network  according  to  the  selected  model.  One  of 
these  is  the  centralized  model  where  a  central 
computing  element  serves  to  all  clients.  Another 
solution  is  the  fully  distributed  approach  in  which 
processing  is  distributed  over  the  computing 
elements  interconnected  via  a  local  area  network. 
Since  the  distributed  model  provides  a  better  fault 
tolerance  for  the  proposed  system,  only  this 
architecture  will  be  emphasized. 


Operator  Consoles 


□  □  □  □ 


Processing  Units 


System  Data  Bus 


5555 


Interface 

Units 

Subsystems 


Figure  5.  Distributed  system  architecture 


The  system,  as  shown  in  Figure  5,  has  different  units 
for  subsystem  integration,  processing  and  operator 
interaction.  The  operator  consoles  provides  user 
interaction  and  graphical  display  of  tactical 
situation.  Displaying  radar  video,  TV  video  and 
digitized  map  are  also  basic  functionalities  presented 
by  the  console  infrastructure.  The  main  processing 
units  provides  computation  power  for  heavy 
processor  loads.  Integration  units  are  used  to  connect 
a  subsystem  to  the  main  system  for  data  transfer. 

Software  Architecture 

The  CPL  software  architecture  as  shown  in  Figure  3 
is  applicable  here.  The  application  for  such  a  system 
can  be  implemented  as  a  CPL  program  on  top  of  the 
RTS.  It  is  also  possible  to  use  some  special  software 
components  to  make  the  system  fault  tolerant.  In 
order  to  keep  the  system  as  highly  available,  the 
number  of  computing  elements  can  be  increased 
letting  them  have  copies  of  active  class  servers  to  be 
used  when  a  new  object  creation  is  necessary.  In  case 
a  node  failure,  a  mechanism  which  detects  the  fault 
can  easily  reallocate  the  active  objects  from  tbe  class 
servers  on  another  node. 

8.2  Implementation 

A  distributed  system  as  described  here  can  be 
implemented  using  the  CORD  System.  The  RTS 
handles  inter-object  communication  and  the  CPL 
provides  a  high  level  interface  to  the  RTS  to  help 
programmers  at  the  language  level.  Entire  C^  system 
can  be  considered  as  one  single  CPL  program  in 
which  all  classes  and  their  instances  are  unique.  This 
program  can  also  be  treated  as  a  domain  which  may 


exist  in  a  naval  C^  system.  A  domain  can  be  defined 
as  a  collection  of  subsystems  and  services.  A  C^ 
system  may  contain  several  domains  that  are 
independent  from  each  other  and  interconnected  via 
special  gateway  software.  Three  typical  domains  can 
be  identified  on  board  a  ship.  Ship  command  level 
domain  owns  all  the  physical  system  resources.  All 
weapons  are  controlled  at  this  level.  Tactical 
command  level  domain  is  managed  by  the  group 
commander,  having  special  tactical  data  links  and 
command  services  controlled  via  operator  consoles. 
A  simulation  domain  can  be  configured  with  the 
existing  consoles  with  simulated  subsystems.  Any 
combination  of  these  domains  can  coexist  in  the 
same  physical  system.  The  CORD  system  provides  a 
feasible  solution  for  implementing  these  domains  by 
using  separate  CPL  programs. 

CPL  device  objects  can  be  used  to  receive  data  from 
external  devices  and  distribute  them  to  the  interested 
active  objects  for  processing.  The  active  objects 
located  on  the  main  computing  elements  process  the 
data  according  to  some  predefined  rules,  the 
command  team  directions  and  other  necessary 
information  previously  distributed  and  stored  in 
passive  objects.  The  processed  data  are  then  either 
fed  into  an  actuator  via  device  objects,  or  distributed 
to  the  entire  system  to  be  stored  and  used  on-demand 
basis.  In  order  to  keep  nodal  passive  data  storage 
always  up-to-date,  each  node  should  have  a  manager 
object  to  subscribe  such  data  and  write  into  the 
passive  objects.  Thus,  the  generation  frequency  of 
data  does  not  effect  the  consuming  clients. 
Whenever  a  client  wants  to  use  data  it  simply  reads 
it  from  the  local  storage,  without  any  context  switch. 

The  core  system  is  identified  as  a  collection  of 
some  major  components  such  as  Platform  Data 
Handling,  Navigation,  Track  Management,  Warfare, 
Subsystem  Management  and  Training.  Following 
sections  describe  how  some  of  these  components  can 
be  implemented. 

Platform  Data  Handling 

This  component  provides  distribution  of  platform 
specific  data  like  reference  time,  compass,  log,  roll, 
pitch,  wind,  global  positioning  system  data  to  all 
nodes  in  the  system  with  appropriate  frequencies. 

Time  synchronization  among  the  nodes  has 
substantial  importance  as  it  is  necessary  for  accurate 
position  calculations  for  weapon  control.  Hardware, 
software  based  solutions  or  a  combination  of  them 
for  network  time  synchronization  has  to  be  used. 

The  publish-subscribe  mechanism  of  the  CORD- 
RTS  provides  an  easy  solution  to  the  distribution  of 
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the  platform  data.  An  active  object,  Local  Database  special  importance  can  be  assigned  priorities  in 

Manager  which  runs  on  each  node  subscribes  to  the  order  to  represent  urgency.  Tracks  are  assigned 

related  methods  of  the  Platform  Data  Generator  priorities  according  to  their  importance  to  the 

running  on  the  integration  unit,  and  stores  the  data  on  platform.  For  instance,  an  engaged  hostile  track  has 

the  local  passive  object  Platform  Database.  Any  the  highest  precedence,  where  a  suspicious  track  has 

active  object  located  on  the  same  node  is  then  able  to  a  higher  priority  than  that  of  a  friendly  track.  The 

access  the  platform  data  with  less  latency.  In  the  objects  manipulating  these  tracks  use  the  priority 

proposed  system,  it  is  sufficient  to  store  the  accurate  values  during  method  calls  and  the  CORD-RTS 

platform  data  only  on  the  main  processing  units.  The  takes  these  values  into  consideration, 

local  databases  provide  manipulated  platform  data  i)  centralized  Model 

such  as  geographical  position,  cartesian  position,  ji  -j  j: 

^  The  centralized  model  provides  central  fusion  of 

drift,  absolute  course  and  speed.  Figure  6  and  Figure  ^  ,  .  .  ■  ■  .  j  r  j-rr  .  t,,. 

„  .  ,  ,  .  , .  ;  .  ■  ,  track  data  originated  from  different  sources.  This 

7  shows  the  design  and  implementation  respectively.  ,  ...  ^  .  r.  . 

°  ^  r  y  approach  utilizes  two  basic  active  objects.  System 

Track  Manager  and  Tactical  Track  Manager,  both 

running  oh  the  main  processing  units  (Figure  8-a). 

The  System  Track  Manager  accepts  primitive  sensor 

tracks  in  relative  position.  When  a  track  source  calls 

this  method  with  a  new  track,  grid  conversion  is 

performed  and  correlation  check  with  the  other 

tracks  is  done.  If  correlation  is  successful,  then  a  the 

primitive  track  data  is  combined  to  the  system  track. 

If  it  is  not  successful,  then  a  new  system  track  is 

Figure  6.  Platform  data  handling  design  constructed.  Every  sensor  track  update  causes  a 

decorrelation  check  inside  the  System  Track 

Track  Management  Manager  which  calls  the  update  method  of  the 

A  system  that  has  to  perform  surveillance  task  Tactical  Track  Manager.  The  Tactical  Track 

must  maintain  a  track  database  as  a  correct  mirror  of  Manager  processes  the  incoming  system  tracks 

external  world.  There  may  be  two  different  design  either  from  the  System  Track  Manager  or  from  Data 

approach  for  track  management.  The  centralized  Link  Manager,  performs  correlation  or  decorrelation 

model  and  the  distributed  model  which  are  shown  in  checks,  maintains  the  Tactical  Track  Database  with 

Figure  8.  In  both  models,  real-time  tracks  that  have  unique  track  identification. 

active  class  Platf oriti_Data_Handler  { 
priority  5; 
static : 

Position_Type  position; 

Stabilization_Data  stab_data; 

Angle_Type  course; 

SpeGd_Type  speed; 

periodic  100: 

Publish_Position ( ) ; 
periodic  50: 

Publish_Stabilization ( ) ; 
periodic  2000: 

Publish_Course_Speed () ; 

publish: 

Update_Pos it ion (Posit ion_Data) ; 

Update„Stabilization(Stab_Data) ; 

Update_Course_Speed(Angle_Type,  Speed_Type) ; 

subscribe : 

Update_Position  to  Serial_Coinm.GPS.Update_Position; 

Update_Stabilization  to 

Serial_Comm.Gyroscope .Update_Stabilization; 

Update_Course_Speed  to 

Serial_Comrn .  Compass .  Update_Course_Speed; 

}; 

class  body  Platf orm_Data_Handler  { 

Publish_Position ( )  { 

Update_Position (position) ; 

) 

Publish_Stabilization ( )  { 

Update„Stabilization {stab_data) ; 

1 

Publish_Course_SpGed ( )  { 

UpdatG_Course_Speed(course,  speed) ; 

) 

_J_ _ 

Figure  7.  Platform  data  handling  implementation 


passive  class  Platform_Database  { 
static: 

Position_Type  position; 

Time_TypG  time_of_update; 

Stabilization_Data  stab_data; 

Angle_Type  course; 

Speed_Type  speed; 

methods : 

Put_Position(in:  Position_Type  Pos) ; 
Get_Position (out :  Position_Type  Pos) ; 
Put_Stab_Data (in:  Stabilization_Data  Stab) ; 
Get_Stab_Data (out :  Stablization_Data  Stab); 
Put_CoursG_Speed (in;  Angle_Type  Crs , 
in:  Speed_Type  Spd) ; 
Get_Course_Speed(out:  Angle_Type  Crs, 
out:  Speed_Type  Spd); 


class  body  Platform_Database  { 

Put_Position(in:  Position_TypG  Pos)  { 

LOCK;  //explicit  control 
position  =  Pos; 

time_of_update  =  CORD_RTS .Get_Current_Time ( ) ; 
UNLOCK; 


Get_Position (out :  Position_Data  Pos)  { 

LOCK; 

Pos  =  position; 

UNLOCK; 

} 

Put_Stab_Data (in:  Stabilization_Data  Stab)  { 
stab_data  =  Stab; 

) 

//  other  methods 
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Figure  8.  Track  management  design 

An  active  object,  Track  Number  Administration, 
provides  unique  track  numbers  whenever  asked.  The 
tactical  tracks  are  updated  system-wide  by  the 
publish  mechanism,  and  stored  locally  as  shown  in 
Figure  8-b.  In  order  to  modify  an  existing  track  or 
delete  a  track,  the  user  calls  the  related  method  of  the 
Tactical  Track  Manager,  which  calls  the  appropriate 
method  of  track  originator  according  to  its  source. 

This  model  provides  a  central  control  over  the  track 
database,  on  the  other  hand,  it  may  cause  a 
bottleneck  at  the  System  Track  Manager  due  to  high 
update  rate  of  track  sources.  Since  correlation/ 
decorrelation  check  is  done  by  one  active  object,  the 
required  processing  has  to  be  handled  by  the  node  on 
which  the  object  resides.  Possible  solutions  are  to 
arrange  the  update  rate  proportional  with  the  track 
priority  and  to  use  computationaly  powerful  nodes. 

2)  Distributed  Model 

The  distributed  model  has  also  two  components. 
Tactical  Track  Manager  and  Source  Handler,  with 
their  associated  track  databases.  The  active  object 
Tactical  Track  Manager  and  the  passive  object 
Tactical  Track  Database  are  located  on  every  node  in 
the  system.  This  component  maintains  a  local 
tactical  track  database.  Tactical  Track  Manager 
object  together  with  the  other  active  object.  Source 
Handler,  form  a  cluster  (Figure  8-c).  A  track  cluster 
is  responsible  from  accepting  raw,  primitive  track 
data  from  the  device  and  converting  to  a  tactical 
track  to  be  sent  out  of  the  node.  Source  Handler 
object  has  to  be  implemented  for  each  type  of  track 
input  device. 

Source  Handler  receives  a  primitive  track  from  the 
track  source  in  relative  position,  performs  the  grid 


conversion  with  respect  to  own  platform  position, 
and  sends  it  to  the  Tactical  Track  Manager.  If  this  is 
a  new  primitive  track  generated  by  the  source,  it  is 
checked  against  all  the  tactical  tracks  in  the  Tactical 
Track  Database  according  to  the  join  criteria.  If  the 
criteria  is  met,  then  the  tracks  are  said  to  be  joined 
and  a  responsibility  check  is  performed  in  order  to 
determine  the  reporting  responsible  source  based  on 
sensor  accuracy.  If  the  track  is  an  update,  then  the 
tactical  track  related  to  that  is  updated  with  the  new 
data.  When  a  new  tactical  track  is  received  from 
another  cluster,  it  is  inserted  into  the  Tactical  Track 
Database  and  updated  thereafter.  If  the  local  source 
is  more  accurate,  then  it  becomes  reporting 
responsible  and  the  Tactical  Track  Manager 
publishes  the  tactical  track  with  information 
provided  by  the  local  source.  If  the  remote  source  is 
more  accurate,  then  the  local  primitive  track  is 
marked  as  a  secondary  source  to  the  current  tactical 
track  held  in  the  Tactical  Track  Database.  Automatic 
take  over  can  happen  if  the  remote  source  ceases 
reporting.  A  central  active  object  is  used  to  provide 
the  unique  track  numbers.  Figure  8-c  shows  the 
design  of  this  approach. 

The  necessary  processing  for  join/disjoin  or 
correlation/decorrelation  checks  is  distributed  over 
the  source  nodes  on  the  network.  If  the  number  of 
tactical  tracks  in  the  system  is  too  high,  then  creating 
a  new  track  causes  a  join  check  to  be  performed  for 
each  track  in  the  database.  In  tbe  worst  case  this  is 
equal  to  the  number  of  existing  tracks.  Since  this 
check  can  be  performed  separately  and  concurrently 
on  each  node,  several  track  tests  can  be  performed  at 
the  same  time,  saving  computation  time.  Moreover, 
disjoin  check  for  every  primitive  update  uses  the 
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processing  power  of  the  node  on  which  the  cluster 
resides.  There  may  be  as  many  track  clusters  as  the 
number  of  track  sources  in  a  system  as  shown  in 
Figure  8-d.  Part  of  the  source  code  of  Track 
Management  is  listed  in  Figure  9. 

Local  Databases 

A  system  requires  objects  residing  on  each  node 
to  be  able  to  access  all  data  fast  enough  to  meet  the 
time  requirements.  For  reducing  the  overhead  due  to 
active  object  method  call  over  the  network,  the 
CORD-RTS  provides  publish-subscribe  mechanism 
which  is  used  for  data  distribution.  However,  some 


objects  may  not  need  to  be  triggered  by  each  new 
instance;  instead,  another  active  object  subscribes  to 
this  kind  of  methods,  accepts  data  and  stores  them  in 
appropriate  databases  which  are  implemented  as 
passive  objects.  It  is  for  sure  that  calling  a  passive 
object  method  is  much  faster  then  calling  a  method 
of  an  active  object.  For  instance,  a  tactical  display 
must  exhibit  all  tracks  with  sufficient  refresh  rate. 
Therefore  each  display  should  have  a  local  track 
database  rather  than  calling  a  central  database 
manager. 

Some  examples  to  the  data  storages  other  than  tracks 


active  class  Track_Management  { 

uses  Track_Number_Handler ,  Track_Database; 

priority  3; 
static : 

int  num_of_tracks; 

methods : 

New_Sensor_Track (in:  Sensor_Track  ST) ; 
Sensor_Track_Update (in:  Sensor_Track  ST) ; 
Wipe_Tactical_Track (in:  Track_Id_Type  Id); 
periodic  2000: 

Send_Tracks ( ) ;  //non-real-time  update  rate 
publish: 

New_Tactical_Track (Tactical_Track) ; 
Tactical_Track_Update (Tactical_Track) ; 
Delete_Tactical_Track (Track_Id_Type) ; 

private : 

Boolean_Type  Join_Track {Sensor_Track&, 

Track_Id_Type£c)  ; 

Boolean_Type  Disjoin_Check(Sensor_Track&) ; 

}; 

class  body  Track_Management  { 

New_Sensor_Track (in:  Sensor_Track£c  ST)  { 
Track_Id_Type  id; 

Tactical_Track  trk; 
if  (Join_Track(ST,  id)  ==  TRUE)  { 

Tactical_Track_DB . Update_KinGtics ( id, 

ST.Get_Kinetics ()); 
Tactical_Track_DB .Get ( id,  trk) ;  ' 

Priority_Type  prio  =  ConvertPrio ( trk) ; 
Tactical_Track_Update (trk,prio) ;  //publish 
}  else  { 

trk  =  Make_Tactical (ST)  ; 
trk . Set_Track_Number ( 

Track_lSJumber_Adinin  .Get_New_Nuinber  ( )  )  ; 
Tactical_Track_DB. Insert (trk) ; 

Priority_Type  prio  =  ConvertPrio(trk); 
New_Tactical_Track(trk,  prio);  //publish 

} 

) 

Sensor_Track_Update (in:  Sensor_Track  ST)  { 
Tactical_Track  trk; 

Track_Id_Type  id; 
if  (Disjoin_Check(ST)  ==  TRUE)  { 

if  {Join_Track(ST,  id)  ==  TRUE)  { 

Tactical_Track_DB .Update_Kinetics (id, 

ST. Get_Kinetics ( ) ) ; 
Tactical_Track_DB.Get (id,  trk) 
Tactical_Track_Update (trk) ;  //publish 
)  else  { 

trk  =  Make_Tactical (ST) ; 
trk . Set_Track_NumbGr ( 

Track_Number_Admin . Get_New_Number ( ) ) ; 
Tactical_Track_DB . Insert ( trk) ; 
Priority_Type  prio  =  ConvertPrio (trk) ; 
New_Tactical_Track(trk,prio) ;  //publish 

} 

}  else  { 

Tactical_Track_DB.Update_Kinetics ( 

id,  ST.Get_KinGtics()); 
Tactical_Track_DB .Get ( id,  trk) ; 
Priority_Type  prio  =  ConvertPrio (trk) ; 
Tactical_Track_Update (trk, prio) ;  //publish 

} 


Wipe_Tactical_Track(in:  Track__Id_Type  Id)  { 
Tactical_Track_DB .Remove ( Id) ; 
Delete_Tactical_Track ( Id) ; 

> 

Boolean_Type  Join_Track  (Sensor_Track5c  ST, 
Track_Id_Type&  Id) 

{  /*  Fusion  algorithm  */  } 

Boolean_Type  Dis join_Check (Sensor_Track&  ST) 

{  /*  Disjoin  algorithm  */  ) 

) 

active  class  Source_Handler  { 
uses  Track_Management; 

priority  4; 
methods : 

Sensor_Track(in:  Track_TypG  T) ; 

Operator_Update (in:  Op_Cmd) ; 
subscribe : 

Sensor_Track  to 

Serial_Comm . nav_radar . Track_Data ;  / /device 

static: 

Track_Management  local_ttm; 

}; 

clas  body  Source_Handler  { 

Sensor_Track(in:  Track_Type  T)  { 

//convert  raw  data  to  sensor  track 
Sensor_Track  trk(T) ; 

if  (trk.Get_Life_Cycle()  ==  NEW_TRACK) 
local_ttm.New_Sensor_Track (trk) ; 
else 

local_ttm. Update_Sensor_Track ( trk) ; 

} 

Operator_Update (in:  Op_Cmd)  { 

//apply  operator  changes  on  tracks 

} 

main: 

//Get  node  name  and  make  a  name:  NodeName 
local_ttm.Bind(NodeName  +  " local_ttm" ) ; 

} 

passive  class  Track_Database  { 
static; 

Tactical_Track  tracks tMAX_TRACKS] ; 

methods : 

Insert (in;  Tactical_Track  T) ; 

Get(in:  Track_Id_Type  Tid,  out;  Tactical_Track  T) ; 
Remove (in;  Track_Id_Type  T) ; 

Update(in:  Tactical_Track  T) ; 

Update_Kinetics (in:  Track_Id_Type  T, 

in;  Kinetics_Type  K) ; 

}, 

class  body  Track_Database  { 

//method  bodies 

} 

device  class  Serial_Comm  { 

read  Track_Type;  //type  to  read  from  device 
buffer  10;  //number  of  elements  in  buffer 

speed  9600; 
methods : 

Read_Data (out :  Track_Type) ; 

publish : 

Track_Data (Track_Type) ; 

}; 


Figure  9.  Track  management  implementation 
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are  platform  position  data,  weapons,  sensors, 
engagements,  tactical  figures,  tactical  commands, 
operators  and  some  state  information. 

Since  CPL  uses  logical  names  to  identify  objects, 
instances  of  the  active  class  Local  Database 
Manager  is  created  on  each  node  using  the  node 
name  as  an  extension  for  identification.  It  can  be 
used  to  create  node  specific  objects  and  subscribes  to 
all  relevant  class  methods  and  write  them  into  the 
appropriate  passive  objects.  Part  of  the  source  code 
is  listed  in  Figure  10. 

Services 

A  service  is  a  graphical  user  interface  (GUI) 
program  that  accepts  user  inputs  or  display  system 
outputs.  These  services  can  also  be  implemented  as 
active  CPL  objects,  provided  that  the  GUI  platform 
works  seamlessly  with  the  CORD-RTS. 

9.  CONCLUSION 

It  is  generally  accepted  that  using  linguistic 
mechanisms  for  abstracting  real-time  constraints  and 

active  class  Local_Database_Manager  { 
uses  Platform_Database,  Track_Database, 
Platform_Data_Handler ; 

priority  3; 
static : 

Name  Node_Name  ; 

Platf  orra_Database  Platf  orro_Database_l  ; 
Track_Database  Tactical_Track_DB_l ; 

methods : 

Local_Database_Manager (in:  Name  Node) ; 

Set_Position (in:  Position_Data) ; 

Set_Stab_Data (in:  Stabilization_Data) ; 
Set_Course_Speed(in:  Angle_Type  Crs, 
in:  Speed_Type  Spd)  ; 

Add_Track (in:  Tactical_Track  T)  ; 

//other  methods  related  to  tracks  ■ 
subscribe : 

Set_Stab_Data  to 

Platform_Data_Handler .Update_Position; 
Set_Stabilization  to 

Pla t  f orm_Data_Handler . Update_Stab_Data ; 
Set_Course_Speed  to 

Plat  f  orm_Da  t  a_Handl e  r . Upda  t  e_C  our  s  e_Spe  ed ; 
Add_Track  to 

Track_Management .New_Tactical_Track; 
Update_Track  to 

Track_Management .Update_Tactical_Track; 
//other  methods  related  to  tracks 

); 

class  body  Local_Database_Manager  { 

Local_Database_Manager (in:  Name  Node)  { 

Node_Name  =  Node; 

) 

Set_Position(in:  Position_Data  Pos)  { 

Platform_Database_l . Put_Position(Pos) ; 

) 

Set_Stab_Data (in:  Stabilization_Data  Stab)  { 
Platform_Database_l . Put_Stab_Data (Stab) ; 

) 

Set_Course_Speed(in:  Angle_Type  Crs, 

in:  Speed_Type  Spd)  { 

Platform_Database_l.Put_Course_Speed(Crs,  Spd) ; 
Add_Track(in:  Tactical_Track  T)  { 
Tactical_Track_DB_l . Insert (T) ; 

) 

//other  method  bodies  related  to  tracks 
main: 

Platform_Database_l.Bind(NodeName  +  "platform"); 
Tactical__Track.__DB_l . Bind (NodeName  +  "trackdb")  ; 

}  _ 


distribution  simplifies  program  design  and 
implementation.  Such  a  capability  even  reduces  the 
need  for  system  level  programming  skill,  enabling 
the  programmer  to  concentrate  on  functional 
behavior  rather  than  complex  low  level 
communication.  For  that  reason,  the  ability  to 
express  distribution  issues  and  real-time  constraints 
through  language  constructs  is  focused. 

It  is  intended  to  use  CPL  to  develop  distributed 
systems  where  soft  real-time  requirements  exist. 
Some  systems  as  presented  or  distributed 
knowledge-based  systems  are  suitable  areas  for 
implementation  in  CPL.  Application  code  written  in 
CPL  can  easily  be  ported  to  new  platforms  after 
adapting  the  CORD  infrastructure. 

This  work  is  still  under  development  at  the  Turkish 
Navy  Software  Development  Center  (TNSDC). 
Research,  implementation  and  test  is  being 
proceeded  adapting  new  system  requirements. 
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1.  SUMMARY 

A  concept  is  proposed  for  an  identity  fusion 
system  to  augment  the  current  positional  data 
fusion  algorithm. 

Data  from  the  aircraft’s  sensors  are  fused  by  an 
Extended  Kalman  Filter  to  develop  tracks  in  an 
air  combat  environment.  The  formed  tracks 
can  be  considered  to  be  a  direct  isomorphic 
mapping  of  the  real  space  entity  onto  a 
software  object.  By  programming  the  software 
object  with  its  own  self  beliefs,  the  ability  to 
vary  its  behaviour  in  response  to  varying 
situations,  a  set  of  goals  and  the  knowledge  of 
how  to  achieve  its  goals,  an  identity 
declaration  can  be  asserted.  This  combines  the 
extendibility  and  reusability  of  a  modular 
architecture  with  the  flexibility  and 
adjustability  of  artificial  intelligence. 

2.  INTRODUCTION 

The  Data  Fusion  Project  at  DERA 
Famborough  aims  to  research  algorithms  for 
combining  sensor  data  for  targets  in  a  way  that 
gives  the  greatest  operational  advantage  to  the 
pilot  in  an  air  combat  situation.  A  complete 
system  must  be  able  to  give  increased 
situational  awareness  of  contextually  relevant 
targets  and  threats,  whether  air  or  ground 
based. 

At  the  sensor  processing  level  the  system  must 
be  able  to  interpret  sensor  data  in  such  a  way 
as  to  track  other  aircraft  and  arrive  at  a  discrete 
hypothesis  regarding  the  aircraft’s  identity. 
The  system  should  also  be  able  to  refine  its 
hypothesis,  in  real  time,  upon  the  injection  of 
new  information.  It  must  also  be  capable  of 
augmenting  the  identity  declaration  with 
information  about  the  environment  and  the 
context  of  the  situation. 


Psychologically,  an  operator  would  use  his 
eyes  to  perceive  the  course  and  identity  of  a 
target  aircraft.  He  would  fuse  the  sensory  input 
and  use  pattern  recognition  to  conclude  that 
the  object  belongs  to  a  certain  group  of  objects, 
e.g.,  the  operator  would  have  narrowed  the 
search  field  down  to  F4,  F15,  F22.  The 
operator  would  then  apply  reasoning  to  each 
alternative  and  select  the  optimal  solution. 

To  reduce  the  workload  of  the  pilot,  the 
computer  would  be  required  to  duplicate  this 
process  by  combining  sensor  data,  a  priori 
knowledge  and  information  regarding  the 
context  of  the  situation,  to  identify  target 
aircraft.  Inference  and  reasoning  techniques 
are  generally  necessary  to  fuse  and  utilise  more 
abstract  data  (i.e.  knowledge  composed  of 
facts  and  heuristics  rather  than  numeric  data). 
It  is  not  possible  to  implement  these  desired 
techniques  using  conventional  statistical 
approaches  and  pure  mathematics. 

Herein  lies  the  motivation  for  applying 
artificial  intelligence  techniques  to  the  problem 
domain. 

This  paper  summarises  the  work  completed  so 
far  on  the  data  fusion  project  at  DERA 
Famborough,  which  consists  of  a  positional 
fusion  algorithm  capable  of  fusing  data  from 
multiple  sensors  and  triangulating  the  targets 
position,  using  kinematic  weaving,  if  in  a 
bearings  only  enviromnent.  Then  an  outline  is 
presented  of  the  potential  augmentation  to  the 
tracker  by  incorporating  the  ability  to  identify 
the  tracks.  The  concepts  that  can  be  applied  in 
order  to  meet  the  new  requirements  are  then 
presented,  together  with  a  conceptual  system 
architecture  that  will  be  coded. 


Paper  presented  at  the  RTO  SCI  Symposium  on  "Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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3.  POSITIONAL  FUSION  AND  THE 
PROJECT  SO  FAR 

The  Sensor  Tracking  Applications  Rig  (STAR) 
‘  comprises  a  suite  of  software  which  run  on  a 
networked  array  of  three  workstations  and 
interfaces  with  the  operator  through  a  set  of 
flight  and  sensor  Hands  On  Throttle  And  Stick 
(HOTAS)  controls.  The  aggregate  system 
allows  man-in-the-loop  assessment  of  data 
fusion  and  tracking  algorithms. 


3.1  Environment 

The  STAR  contains  a  range  of  data  files 
containing  specifications  for  aircraft  models, 
scenarios,  sensors  and  other  systems.  These  are 
read  at  mn  time  and  define  the  system 
behaviour  during  a  real-time  mn.  Scenarios 
may  be  defined  as  being  “canned”,  that  is  with 
all  the  players’  movements  defined  in  a 
scenario  file.  Alternatively,  the  motion  of  one 
or  more  aircraft  may  be  controlled  by  the 
operators  flight  controls. 


3.2  Program 

Sensor  measurements  are  supplied  to  data 
fusion  and  tracking  algorithms  and  the 
resulting  estimated  tracks  are  displayed  on  a 
generic  B-scope  and  Tactical  Evaluation 
Display.  Measurements  may  either  be 
generated  in  real-time  by  the  software,  or  be 
read  from  file.  In  the  latter  case  the  file  may 
have  been  generated  by  previous  real-time 
activity  or  recorded  in-flight  (real  world)  data 
may  be  used.  An  interactive  ‘gods-eye’  view 
of  the  scenario  displays  the  ‘hue’  behaviour,  if 
available  in  the  case  of  real  data,  of  all 
platforms. 


The  hacker  uses  sequential  estimation  for 
multisensor  positional  data  fusion.  An 
Extended  Kalman  Filter  (EKF)  (as  in  Figure  1) 
implements  kinematic  ranging  to  passively 
hiangulate  the  target  aircraft’s  position.  There 
is  an  option  of  implementing  either  cenhalised 
or  autonomous  fusion.  The  autonomous  fusion 
algorithm  will  beeome  more  useful  as  sensors 
become  more  intelligent  with  an  increased 
capability  of  developing  local  hacks  prior  to 
fusion. 
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Figure  2  -  ARCHITECTURE  FOR  Feature  Level  Multisensor  Identity  Fusion 


4.  IDENTITY  FUSION 

Determination  of  track  identity  or  identity  fusion 
is  concerned  with  attempting  to  convert  multiple 
sensor  observations  and  uncertainty  of  a  target’s 
attributes  (parametric  sensor  data)  into  a 
consensual  declaration  of  identity.  The  model  for 
feature  level  fusion  is  illustrated  in  Figure  2.  From 
a  software  engineering  standpoint,  there  exists 
formal  frameworks  into  which  specific  techniques 
can  be  inserted  and  any  system  can  be  represented, 
checked  and  verified.^ 

Referring  to  Figure  2,  each  sensor  provides 
observational  data  from  which  feature  vectors  can 
be  extracted.  The  feature  vectors  are  then  aligned 
in  space  and  time.  The  aligned  vectors  are  then 
associated  with  other  measurements  and  pre¬ 
existing  tracks.  These  feature  vectors  are  then 
fused  together  into  a  single  feature  vector,  which 
in  turn  is  input  to  an  identity  declaration  technique 
such  as  a  neural  network  or  clustering  algorithms. 
The  output  is  then  a  consensual  declaration  of 
identity  based  on  the  combined  feature  vectors 
from  all  of  the  sensors.^ 

5.  POTENTIAL  AUGMENTATION  OF  THE 
TRACKER  BY  IDENTIFYING  THE 
TRACKS 

Next  is  a  discussion  of  what  further  requirements 
exist  to  enable  the  fused  tracks  formed  by  the 
Extended  Kalman  filter  tracking  algorithm,  to  be 
identified.  The  requirements  extend  beyond  basic 
reasoning  techniques  and  toward  intelligent 


techniques  capable  of  inference,  adaptation 
and  machine  learning. 

The  project  is  ongoing  with  potential  for 
expansion.  Duplicating  pre-written  code  is 
unnecessary  and  should  be  avoided.  It 
would  be  useful  if  the  code  were  reusable. 
Positional  fusion  and  identity  fusion  are 
low  level  components  of  a  much  wider 
field.  The  system  must  be  able  to  integrate 
and  combine  easily  integration  in  whole  or 
in  part  with  existing  legacy  systems.  It 
needs  to  be  extendible  and  compatible. 

The  system  needs  to  be  understandable  so 
that  future  generations  can  take  on  the 
development,  maintain  the  software  and 
build  on  what  has  been  achieved.  There  is 
great  potential  for  the  problem  domain  to 
get  unnecessarily  complicated,  so 
decomposing  the  problem  into  several 
modules  will  simplify  the  system.  Further 
benefits  of  decomposing  the  problem  into 
modules  are  that  errors  or  abnormal 
conditions  are  less  likely  to  effect  the 
system  its  entirety,  thus  making  it  more 
robust. 

In  order  to  meet  these  criteria,  the 
principles  of  object-orientation  together 
with  selected  artificial  intelligence 
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techniques,  can  be  applied  to  the  problem  domain. 

6.  CONCEPTS  THAT  CAN  BE  APPLIED  IN 
ORDER  TO  MEET  THE  REQUIREMENTS 

There  exists  an  entity  of  interest  in  real  space.  The 
sensor  returns  are  such  that  the  integrated  tracking 
algorithm  has  formed  a  track  and  has  a  range  and 
bearing  estimate  on  the  entity.  The  entity  in  real 
space  therefore  has  a  direct  isomorphic  mapping 
onto  the  data  fusion  system.  The  associated  object 
within  the  data  fusion  system  can  be  created  to  be 
pro-active  and  autonomous,  with  it’s  own  beliefs, 
desires  and  intentions.  It  can  be  capable  of 
intrinsic  reasoning,  ''  have  the  ability  to  query 
segregate  knowledge  sources  ^  and  have  a  goal  or 
existential  purpose  to  identify  itself 

6.1  Objects 

The  real-space  entity  representing  an  air  platform 
can  be  considered  to  have  a  software  counterpart 
called  an  object.  An  object  is  an  intuitive  mapping 
of  the  problem  space  to  a  computational  model.® 
Objects  can  contain  data,  knowledge  bases 
(properties)  and  procedures  (events),  objects 
inherit  properties  from  parent  objects  based  on  an 
explicitly  represented  object  hierarchy  and  objects 
can  communicate  with,  as  well  as  eontrol  other 
objects. 

6.2  Agents 

An  agent  may  be  thought  of  as  a  meta-object,  they 
too  have  the  properties  of  objects.  Unlike  objects 
however,  agents  have  goals  and  pro-actively  work 
to  achieve  them.  ^  An  agent  is  described  as  a  self- 
contained,  interactive  and  concurrently  executing 
object.^  An  objects  methods  are  simply 
procedures  coded  by  a  programmer  to  deal  with 
different  messages  (e.g.  A  dog  can  bark  when 
pinched).  An  agent,  on  the  other  hand,  can 
synthesise  plans  to  deal  with  contingencies  not 
foreseen  by  the  programmer,  and  can  learn  from 
experience  (e.g.  A  dog  agent  can  discover  that 
insistent  barking  gets  him  fed  a  lot  quicker). 

If  programmed  as  such,  agents  can  embody 
cognitive  notions.  These  are  generally  designated 
the  terms  ‘Beliefs,  Desires  and  Intentions’  (BDI) 
and  determine  how  rational  agents  will  act.  Agents 
can  be  defined  as  entities  whose  state  is  viewed  as 
consisting  of  mental  components  such  as  beliefs, 
capabilities,  choices,  and  commitments  ...  What 
makes  any  hardware  or  software  component  an 
agent  is  precisely  the  fact  that  one  has  chosen  to 
analyse  and  control  it  in  these  mental  terms.  ®  This 
leads  to  the  study,  modelling  and  specification  of 
mental  attitudes.  An  accomplishment  that  stand¬ 
alone  statistical  analysis  is  not  capable  of 


The  agents  within  the  architecture  will  be 
composed  of  interacting  task-based 
components.  The  components  can  be 
primitive  reasoning  components  using  a 
knowledge  base,  but  may  also  be 
subsystems  which  are  capable  of 
performing  tasks  using  methods  as  diverse 
as  neural  networks,  bayesian  inferencing 
and  genetic  algorithms. 

The  value  of  an  intelligent  agent  will 
typically  be  as  an  augmentation  upon  the 
value  of  the  application  that  it  enhances. 
The  resulting  system  will  consist  of  highly 
modular,  pluggable  components  for 
inferencing,  authoring  and  communication, 
embodied  in  a  fine-grained  object-oriented 
class  library.  '*  The  resultant  agent  will 
therefore  be  capable  of  reasoning  about  its 
own  tasks,  processes  and  plans,  its 
knowledge  of  other  agents,  its 
communication  with  other  agents,  its 
knowledge  of  the  world  and  its  interaction 
with  the  world.  " 

7.  LOWER  LEVEL  DESCRIPTION 

Agents  possess  knowledge  of  the  various 
information  sources  and  how  to  access 
them,  and  an  acquaintance  model 
specifying  the  abilities  of  other  information 
agents.  The  intelligent  agent  would  consist 
of  three  sections,  the  Self  model,  the  World 
model  and  the  Acquaintance  model. 

The  self  model  is  a  database  of 
characteristics  and  properties,  together  with 
confidences.  It  would  also  contain  the 
agent’s  goals  and  intentions.  An  example  is 
shown  in  Figure  3.  The  world  model 
represents  the  real  world.  This  would  be  the 
intelligence  database  or  knowledge  base. 
However,  a  full,  shared  situational  model 
would  be  too  complex.  Therefore  and  agent 
specific  world  model  can  be  defined 
whereby  the  part  of  the  real  world  relevant 
to  that  agent  constitutes  the  agents  world 
model.  The  acquaintance  model  is  a  list  of 
the  outputs  given  and  inputs  needed  by 
other  agents  and  the  inputs  and  outputs 
that  they  take.  This  enables  communication 
between  modules  within  the  architecture. 
For  example,  the  operational  agent  might 
want  to  communicate  with  other 
information  agents  in  order  to  prevent 
unnecessary  duplication. 
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Self  model  -  requirements,  internal  state  of  the  agent 

Acquaintance  model  -  who  the  agent  can 
communicate  with  and  how 

Characteristic 

Value 

Hypothesis  1 

Confidence  1 

Acquaintances 

Size 

30 

Small 

16% 

1 

Names/locations  of  other  agents 

IFF 

friend 

Objects  1-7 

75% 

2 

What  other  agents  require  as  input 

JEM 

Emitter  3 

61% 

3 

What  other  agents  output 

PRI 

280 

Emitter  2 

82% 

■ 

How  to  reformat  for  cross 
comprehension 

World  Model  - 
to  the  agent 

representation  of  the  system  according 

Ability  model  -  what  the  agent  can  do 
with  its  information 

Object 

Confidence 

World 

Abilities 

Air  target 

90% 

Fuel  flow  graphs 

1 

Query  intelligence  database 

Fixed  wing 

86% 

Environmental  effects 

2 

Self  terminate 

Harrier 

58% 

Altitude  effects 

3 

Maintain  and  store  track 

Tornado 

40% 

Air  target  capabilities 

4 

Update  reasoning 

5 

Hierarchical  search 

Figure  3  -  Information  Agent  Structure.  Numerical  values  are  arbitrary  examples 

AND  NOT  REAL  DATA. 


As  the  model  becomes  more  sophisticated  and 
the  individual  agent  integrates  into  the  society 
that  forms  the  complete  system,  the  emergent 
properties  that  ought  to  become  evident  are  as 
follows; 

•  Autonomy  -  Agents  exercise  control  over 
their  internal  state  and  their  behaviour. 

•  Pro-activity  -  The  ability  to  execute 
autonomously  without  being  evoked 
externally. 

•  Reactivity  -  The  ability  of  the  agent  to 
perceive  and  react  to  changes  in  the 
environment. 

•  Communication  -  The  ability  to  exchange 
information  with  other  entities  such  as 
other  agents,  objects,  humans  and  the 
environment. 

•  Reasoning  -  The  ability  to  infer  and 
extrapolate  based  on  current  knowledge, 
information  and  experiences. 

•  Planning  -  An  agent  can  synthesise  and  ' 
choose  between  different  courses  of  action 
with  an  aim  to  achieving  its  goals. 


•  Learning  -  An  agent  may  be  able  to 
accumulate  knowledge  to  assemble  a 
history  and  so  adapt  its  behaviour  to 
respond  to  new  situations. 

The  main  problems  with  building  agent 
systems  is  that  they  have  all  the  problems 
associated  with  traditional  distributed, 
concurrent  systems,  and  have  the  additional 
difficulties  which  arise  from  having  flexible 
and  sophisticated  interactions  between 
autonomous  problem  solving  components. 

There  are  also  social  and  psychological 
impediments  to  the  implementation  of  agent 
systems.  The  operator  must  be  comfortable 
delegating  tasks  to  the  computer.  The  stability 
and  consistency  of  decision  is  also  important, 
if  the  system  frequently  changes  its  mind  then 
the  operator  will  lose  confidence  in  it  and  not 
use  it.  It  must  demonstrate  reliability  and 
consistency. 
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8.  CONCEPTUAL  SYSTEM 
ARCHITECTURE 

If  a  software  project  has  been  a  long  time  in 
development  then  it  often  reaches  a  level  of 
complexity  such  that  rewriting  the  project  is 
unfeasible  and  impractical.  That  is  one  reason 
why  many  industrial  agent  applications  are 
additions  to  well  developed  existing  systems 
(legacy  systems).  A  reasonable  way  to  connect 
the  legacy  system  to  the  new  system  is  to 
encapsulate  it  as  an  agent. 


Building  an  ‘agent  wrapper’  around  the 
multisensor  data  fusion  tracking  algorithm  will 
allow  for  the  intercormection  and 
interoperation  of  the  tracker  with  the  identity 
fusion  architecture.  The  benefits  of  this 
application  integration  are  that  legacy  systems 
can  still  remain  useful  and  be  exploited  by 
other  pieces  of  software. 


Figure  4  -  Optimised  System  Architecture 
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9.  CONCLUSIONS 

A  complete  multisensor  data  fusion  system 
must  be  able  to  give  increased  situational 
awareness  of  contextually  relevant  targets  and 
threats. 

At  a  low  level  the  system  must  be  able  to 
process,  fuse  and  explicate  all  incoming 
information,  whether  this  is  from  the  sensor 
suite,  the  a  priori  information  database  or  any 
intelligent  support.  The  low  level  data  fusion 
system  should  be  able  to  utilise  this 
information  to  efficiently  and  accurately  track 
and  identify  other  aircraft.  The  stability  of 
decision  is  also  important,  the  decision  must  be 
consistent  and  not  fluctuate. 

Currently,  the  project  consists  of  an  Extended 
Kalman  Filter  tracker  which  implements 
kinematic  ranging  to  passively  triangulate  the 
target  aircraft’s  position.  The  tracker  has  been 
validated  against  authentic  flight  data  and  has 
demonstrated  robustness,  accuracy  and 
proficiently  meets  specified  performance 
criteria.  It  is  capable  of  multisensor  data  fusion 
in  either  a  centralised  or  autonomous  mode. 

Present  research  is  concerned  with  determining 
the  identity  of  the  formed  tracks. 

Arguably,  the  best  method  for  identifying 
targets  and  reasoning  about  the  situation  in  a 
combat  environment,  is  the  cognitive  ability  of 
the  pilots.  The  demands  on  the  pilot  are 
increasing  with  the  sophistication  of  the 
aircraft.  The  workload  of  the  pilot  needs  to  be 
reduced  as  much  as  possible,  so  by 
incorporating  cognitive  notions  and  artificial 
intelligence  into  the  system  itself,  more  tasks 
can  be  carried  out  by  the  aircraft’s  system. 
There  is  also  potential  for  the  automated 
system  to  detract  from  the  fallibility  of  the 
pilot  and  increase  the  speed  of  response. 

By  building  an  intelligent  agent  wrapper 
around  the  tracking  algorithm  an  incorporating 
classification  techniques  the  present 
assemblage  can  be  integrated  into  a  data  fusion 
environment,  that  can  provide  a 
comprehensive  decision  support  to  the  pilot,  by 
giving  him  an  enhanced  awareness  of  the 
situation,  in  an  air  combat  situation. 
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Summary 

MIDS-Triangulation  denotes  a  trianguiation  procedure 
to  determine  the  range  of  ownship  spokes  by  means 
of  spokes  and/or  kinematically  ranged  tracks  of  other 
platforms.  It  is  a  method,  which  is  especially  suitable 
to  determine  the  position  of  jammers. 

This  paper  develops  a  method  to  determine  in  three 
steps  the  range  of  a  target/spoke  to  the  own  aircraft. 
The  first  step  consists  in  the  calculation  of  the 
geometric  intersection  points.  In  the  next  step,  the 
geometric  intersection  points  will  be  investigated 
whether  they  behave  as  real  existing  targets  by  means 
of  physical  parameters.  To  this  purpose  an  evaluation 
function  will  be  introduced,  indicating  the  measure  of 
believe  for  an  existing  and  therefore  known  target, 
which  behaves  like  a  calculated  geometric 
intersection  point. 

The  set  of  believable  intersection  points  will  then  be 
tested  on  association  of  ownship  spokes  with  MIDS 
spokes.  The  association  test  (or  de-ghosting  test) 
decides  whether  an  ownship  spoke  can  be  determined 
unambiguously  or  not.  Crucial  for  the  hit  rate  of 
successful  trianguiation  is  the  geometric  constellation 
of  the  MIDS  platforms  to  the  ownship. 

Simulation  results  prove  the  correctness  and 
efficiency  of  the  developed  method. 

Keywords 

Trianguiation,  Deghosting,  Association,  Fuzzy  sets. 
Fuzzy  Logic,  Spokes 

List  of  Symbols 

(j)  Geographic  Latitude 

X  Geographic  Longitude 

Main  axis  of  the  earth  ellipsoid 
e  Excentricity  of  the  earth  ellipsoid 

h  Height  above  the  earth  ellipsoid 

(p  Spoke  azimuth  angle 

£  Spoke  elevation  angle 

R'  Distance  of  Spoke  Intersection  point  to 

the  MIDS  platform 


R 

^.i 

Distance  of  Spoke  Intersection  point  to 
the  Ownship 

3-dimensional  distance  vector  from 

MIDS  platform  to  the  ownship 

Oj 

3-dimensional  vector  of  the  ownship 

spoke 

3-dimensional  vector  of  the  MIDS 
platform  spoke 

,  A  ,h  )  3-dimensional  position  vector  of  the 

ownship,  expressing  the  relationship 
between  the  geographic  co-ordinates 
height  hj above  ground  and 

the  elliptical  co-ordinates  of  the 
ownship  position  above  the  earth 

ellipsoid. 

3-dimensional  position  vector  of  the 


MIDS  platform  expressing  the 
relationship  between  the  geographic  co¬ 
ordinates  ,  A/  )  .height  /i, '  above 
ground  and  the  elliptical  co-ordinates  of 
the  MIDS  platform  position  above  the 
earth  ellipsoid 


Ownship  velocity  in  (N,E,D)  system  of 
the  ownship  platform 

Wj 

Ownship  velocity  in  (N,E,D)system  of 
the  MIDS  platform 

DitpA) 

Transformation-matrix,  to  transform  a 
point  given  in  elliptical 

coordinates  to  {N,E,D)  Local  Horizontal 
Coordinates 

1  introduction 

MIDS  stands  for  Multi-functional  Information 
Distribution  System  and  is  a  time  division  multiple 
access  data  link,  permitting  data  exchange  among 
airborne,  land  and  sea  forces,  and  any  combination  of 
these  armed  forces.  The  data  link  is  currently 
introduced  by  NATO  nations. 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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This  treatise  suggests  a  triangulation  and  de-ghosting 
method  for  supposed  air-to-air  targets  reported  as 
spokes  (direction  information  only:  azimuth,  elevation) 
by  the  ownship  platform  and  other  MIDS  platforms. 
The  aim  of  this  paper  is  to  develop  a  concept,  which 
can  be  easily  implemented  in  any  airborne  computer, 
in  order  to 

•  support  jammed  onboard  sensors  by  providing 
them  with  calculated  ranges  and  range  rates,  in 
order  to  locate  jammers 

•  provide  this  data  for  the  displays  in  the  cockpit  to 
improve  the  situation  awareness 

•  provide  this  data  to  a  possibly  available  kinematic 
Sensor  Fusion  process  in  a  fighter/bomber 
aircraft. 


P  on  the  earth  ellipsoid.  The  angle  is  positive  when  the 
radius  vector  points  to  a  point  on  the  north 
hemisphere  on  the  ellipsoid.  The  latitude  and  the 
longitude  of  this  coordinate  system  are  called 
geocentric  coordinates. 

Note:  The  geocentric  latitude  is  not  identical  with  the 
geographic  latitude. 

2.1.2  Geographic  Coordinates 
The  geographic  latitude  (p  is  the  angle  between  the 
Normal  of  the  earth  ellipsoid  and  the  equatorial  plane 
of  the  earth  ellipsoid. 

The  geographic  longitude  A  is  the  angle  between 
the  Greenwich  Meridian  and  the  Meridian  of  a  point 
P  on  the  earth  ellipsoid,  measured  in  east  direction. 


At  the  beginning,  all  necessary  terms  and  data  are 
defined  and/or  explained,  which  are  frequently  used 
in  the  succeeding  chapters. 

Chapter  5  starts  with  the  preprocessing  of  the  spoke 
data,  e.g.  time  alignment  of  the  spoke  data  to  a 
common  point  in  time  and  transformation  of  all  spoke 
data  in  a  common  reference  frame  prior  to  start  with 
the  calculation  of  the  slant  range  and  slant  range  rate. 
The  slant  range  and  slant  range  rate  for  the  ownship 
spokes  are  calculated  relative  to  the  ownship  platform 
and  it  can  be  shown  that  the  slant  range  relative  to 
the  MIDS  platform  can  also  be  calculated,  however  in 
this  case  the  range  is  dependent  from  the  unknown 
height  of  the  MIDS  platform. 

Chapter  6  introduces  the  motivation  for  the 
development  of  the  credibility  tests  and  defines  a  set 
of  credibility  measures  expressing  the  degree  of 
strongness  that  a  target  belongs  to  the  fuzzy  set, 
which  is  defined  as  the  set  of  realistic  target  behavior. 

Chapter  7  presents  two  alternatives  for  the 
assignment  of  MIDS  spokes  to  an  ownship  spoke. 

Chapter  8  with  the  simulation  results  complete  the 
paper.  The  simulation  results  are  very  encouraging. 
The  authors  believe  that  with  more  improved 
computer  hardware  in  terms  of  processing  power,  an 
improved  MIDS  data  link,  and  approximations 
minimizing  the  calculation  effort  for  the  slant  range 
and  slant  range  rate  calculation,  this  triangulation 
method  can  be  very  successful  in  real  applications. 

2  Definitions 

2. 1  Used  Coordinate  Systems 

2.1.1  Earth  fixed  Coordinate  System 

The  coordinates  {x,y,z;0)  are  noted  as  earth  fixed 
coordinates  with  the  origin  0  in  the  mass  center  in  the 
earth.  The  x-Axis  is  in  the  equator  plane  and  parallel  to 
the  Meridian  through  Greenwich  0°,  the  z-Axis  is 
orthogonal  to  the  x-Axis  and  directed  to  the  North 
pole,  and  the  y-Axis  orthogonal  to  the  (x,z)-plane,  but 
directed  to  the  east.  The  angle  A  is  called  the 
Longitude  and  is  measured  from  Greenwich  Meridian 
0°  to  the  east.  The  latitude  is  the  angle  measured 
from  the  equator  plane  to  the  Radius  vector  of  a  point 


2. 1 .3  (N,E,D)  Coordinate  System 

The  (NED)  coordinate  system  is  defined  to  be  an  earth 
fixed  frame  with  the  origin  in  the  aircraft.  Down,  D,  is 
defined  to  be  the  normal  to  the  reference  ellipsoid 
which  is  an  approximation  of  the  geoid.  The  north 
axis,  N,  is  in  the  direction  of  the  projection  of  the 
earth’s  inertial  angular  velocity  vector  into  the  local 
horizontal  plane  (the  plane  which  is  perpendicular  to 
down  direction).  The  east  direction,  E,  is  orthogonal  to 
N  and  D.  The  frame  is  right-handed. 

2.2  Definition  of  the  Polar  Coordinates  of  a 
point  in  the  Local  Horizontal  SvstemfNED) 

For  the  formulation  of  the  intersection  point  equations 
of  a  spoke  -  line,  it  is  required  to  define  the  direction 
angles  of  a  supposed  target  relative  to  the  ownship 
position. 

Azimuth  angle  clockwise  positive  between  the 
North  axis  and  the  projection  of  the  spoke  line  on 
to  the  (N,E)-  plane 

6:  Elevation  angle  is  the  angle  between  the  spoke¬ 
line  and  the  projection  of  the  spoke-line  onto  the 
(N,E)-plane.  The  angle  is  positive,  when  the 
spoke-line  directing  to  the  target  is  above  the 
(N,E)  -  plane. 

r  Distance  between  the  origin  of  the  NED  frame  of 
a  system  and  the  target 


The  relation  between  the  polar  coordinates 
(^,£,r)and  the  cartesian  coordinates  (N,E,D)  of  a 
point  P  in  space  is  given  by  the  following  equation: 

Equation  2.2-1 


P  = 


^COS  £  •  COS  (p  ' 

r  cos  e  •  sin  (p 


sin  e 


J 
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P(x_N  ,  x_E,  x_D  ) 


Figure  2.2-1  Definition  of  Spoke  Coordinates  in 
the  NED  frame 


Although  the  Stanag  message  doesn’t  contain  height 
information  of  the  MIDS  platform  and  of  the  spoke 
(e.g.  in  form  of  an  elevation  angle),  these  parameters 
shall  be  amended: 

*  MIDS  Spoke  Elevation:  f , ' 

*  MIDS  Spoke  Elevation  Rate: /9,.' 

*  MIDS  Altitude: /l, ' 

*  MIDS  Vertical  Velocity:  ' 

None  of  these  four  parameters  are  contained  in  the 
MIDS  message,  but  it  will  be  shown  that  the  elevation 
and  the  elevation  rate  can  be  reconstructed,  provided 
one  can  make  some  assumptions  about  the  MIDS 
altitude,  e.g.  /i/=  10,000m.  It  will  be  seen  that  the 
knowledge  of  the  MIDS  Vertical  Velocity  is  in  fact  not 
required. 


2.3  Definition  of  a  ‘Spoke’ 

If  one  knows  only  the  direction  of  a  point  P,  i.e.  the 
angles  (p  and  £ ,  but  not  the  relative  distance 
/■between  the  origin  in  the  ownship  (N,E,D) 
coordinate  system  and  the  supposed  target,  then  one 
notes  this  geometric  relation  as  a  ‘Spoke’. 


3  MIDS  Spoke  Data 

MIDS  spokes  are  broadcasted  every  six  seconds  in  the 
network  by  means  of  a  dedicated  message  of  the 
Stanag  5516  and  can  be  received  by  all  network 
participants.  The  spoke  data  are  refreshed  with  a 
frequency  of  1  /6  Hz.  However,  there  is  no  indication 
in  the  message,  which  permits  to  identify  a  specific 
spoke  in  two  consecutive  transmissions. 

For  the  development  of  the  triangulation  method  it  will 
be  assumed  that  for  the  triangulation  the  data  sets  of 
N  spokes  5, ,...,5^  are  available.  The  parameters  of 
the  MIDS  spokes  shall  be  noted  as  follows: 

*  MIDS  Platform  Longitude:  ' 

*  MIDS  Platform  Latitude: 

*  MIDS  Platform  Speed:  v/ 

*  MIDS  Platform  Course: 

*  Spoke  Azimuth:  ' 

*  Spoke  Azimuth  Rate:  OJ^ ' 

*  Spoke  Range  Estimate:  r, ' 

*  Spoke  Range  Accuracy:  Af) ' 

*  Time  Tag:  ' 


Consequently  a  spoke  is  characterised  by  the  tupel 

s,={a;  4^  ,v; ,(p;  ,(o;  ,r;  ,Ar;  ,t;) 


4  Ownship  Spoke  Data 

It  will  be  assumed  that  for  the  Triangulation  the  data 
of  M  ownship  spokes  are  available.  The 

parameters  of  the  ownship  spokes  will  be  noted  as 
follows: 

*  Ownship  Platform  Longitude:  A  ^ 

*  Ownship  Platform  Latitude:  (jij 

*  Ownship  Platform  Altitude:  hj 

*  Ownship  Velocity: 

*  Ownship  Spoke  Azimuth: 

*  Ownship  Spoke  Azimuth  Rate:  (Oj 

*  Ownship  Spoke  Elevation:  Ej 

*  Ownship  Spoke  Elevation  Rate:  Pj 

*  Ownship  Time  Tag:  tj 

An  ownship  spoke  is  therefore  characterized  by  the 
tupel  Oj  =  (Aj , </)j , hj ,  Vj , (Pj ,COj,£j,  Pj , tj  ) 
whereby 

Vj  =  {pf^  j  j  ,V,j  j  is  a  column  vector. 

5  The  Triangulation  Mathematics 

For  the  triangulation  calculations  it  will  be  assumed 
that  all  the  information  about  a  spoke  is  available  from 
the  MIDS  spoke  message.  In  order  to  be  able  to 
determine  the  three-dimensional  position  and  velocity 
of  a  supposed  target,  only  the  slant  range  and  the 
slant  range  rate  in  relation  to  the  ownship  position  is 
required.  The  ownship  spoke  is  then  completely 
described  as  state  vector  in  the  polar  coordinate 
system  by  means  of  the  azimuth  angle,  the  azimuth 
angle  velocity,  the  elevation  angle  and  the  elevation 
angle  velocity. 


Note:  All  MIDS  spoke  variables  are  going  to  be  noted 
as  dashed  variables  ('). 


5. 1  Extrapolation  of  the  Ownship  Data 

The  position  of  the  ownship  spokes  and  of  the 
ownship  are  going  to  be  extrapolated  to  the  point  in 
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time  of  the  validity  of  the  MIDS  spokes.  This 
proceeding  is  highly  recommended  since  there  is 
more  information  available  from  the  ownship 
spokes(Elevation  and  3-dimensional  velocity). 


Equations  5.1-1 


hj=hj 

f/  =(Pj+iti'-tj)-C0j 
£j=ej+(i;-tj)‘Pj 


Unless  otherwise  stated,  it  will  be  assumed  that  in  the 
following  chapters  the  above  parameters  are  already 
extrapolated  according  to  Equations  5.1-1  and 
therefore  the  tilde  indices  will  not  be  used.  This  eases 
the  reading  of  the  paper. 

Angular  rates  from  azimuth  and  elevation,  as  well  as 
the  ownship  velocities  will  not  be  extrapolated. 

5.2  Derivation  of  the  Intersection  Equations 

5.2.1  Determination  of  the  Basis-vector  from  the 
MiDS  Piatform  to  the  Ownship 

Figure  5.2-1  sketches  the  triangulation  method.  In  the 
following  it  will  be  assumed  that  the  positions  of  the 
ownship  and  the  MIDS  platforms  are  available  in 
(N,E,D)  coordinates  and  that  the  kinematic  data  of  the 
ownship  platform  are  already  extrapolated  according 
to  Equations  5.1-1.  The  extrapolated  ownship  data  are 
going  to  be  transformed  into  the  local  horizontal 
coordinate  system  of  the  MIDS  platform  with 
origin(0,0,0). 


Ownship 

MIDS  piatform 

Figure  5.2-1  Geometric  Basis  for  Triangulation 

The  distance  vector  Xj{  see  Figure  5.2-1)  between 

the  MiDS  platform  and  the  ownship  platform  has  in 
the  (N,E,D)  system  of  the  MIDS  platform  the  following 
form: 

Equation  5.2-1 


whereby 


^ -siri<l)l<osXl  -sin^'-sinX/  cos<p/ 
-sin^'  cosAI  0 
-cos^J<osXI  -cos^-sinXi'  -singly 


R. 


■  +  hi)-cos(pj  cos  A, 


-sin  (pj 

R.  .  . 

i  L  ,  .  J-T  ^  ‘ 

<t>j 

(•7=1=^ 

yi-G  -Sin  <l)j 


R, 


■+  h  ,’-cos0  ,.'cosA,  ' 


^  -sin^ 

(  I  .^*  T —  +  /i,')-cos<l,'siiiA,' 


2  •  2 

€  -sm 


'(.(i-e’) 


:  -F  /i/  )  •  sin 


The  computation  of  Equation  5.2-1  yields 

Equation  5.2-2 

•%7  -{N J¥h^cos<}lsin(liJ-si^^<|>'^cos(|>f 
cos{X■-XJ)y-  ^  -cos^l'Af^ 

Xj;j  =  (N  j+hj)(cos</>jSin(X;-Xj); 

Xoj  -  j^h]icos(l>]cos<l>fo^X\-X) + 
sin^lsirKpi)- ^  -siiKpl-tK-N ^-h' 


with 


n:= 

Nj  = 


Re 

■yjl-  ^  -sin^ 

K 


AK  =  Nj  ■  sin  -  TV,  '•  sin  ' 
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5.2.2  Transformation  of  the  Ownship  Velocity 
into  the  MiDS  Piatform  NED-Coordinate 
System: 

The  relative  velocity  of  the  ownship  as  measured  from 
the  MIDS  platform  yields 


Equation  5.2-3 


w 


NJ 


Define 


\ 


Then  Equation  5.2-3  can  be  written  as 

Equation  5.2-4 


5.2.3  Intersection  Point  of  the  Ownship  Spoke 
with  the  MiDS  Spoke 

The  direction  vector  Oj  of  the  ownship  RADAR  spoke 
Of  in  the  MIDS  platform  is  given  by  the  transfor¬ 
mation  equation 


Equation  5.2-5 


^ O'  ^ 

o\ 


EJ 


with  Oj  = 


^COS£j  •COS(Pj  ^ 

cosEj '  sirup j 
-sine, 


being  the  direction  of  the 


spoke  in  the  ownship  (NED)  coordinate  frame.  The 
direction  vector  of  a  MiDS  spoke  5,  is  defined  as 


Equation  5.2-6 


‘^N,i 

cose  -  CO sq)  r 

= 

cose  j'-si  n<p  ■ 

With  the  above  given  definitions  and  designations,  we 
can  now  establish  the  intersection  equations  for  the 
linear  equations  of  the  MIDS  and  ownship  spokes: 


Equation  5.2-7 

R'-S^  =Xj+R-d'j 


5.2.4  Spoke  Velocities  in  the  MIDS  Platform 
(NED)  Coordinate  System 

By  differentiation  -  with  respect  to  time  -  of  the  spoke 

vectors  O'^and  S  given  in  Equation  5.2-5  and 
Equation  5.2-6  above,  one  yields  the  spoke  velocities: 


Equation  5.2-8 


^sine)-cos^'] 

-CJj- 

cospj 

-Pr 

sine)-sinp^ 

0 

^  7 

cosf, 

^  ■'  J-i 

Equation  5.2-9 


^sin£/-cos^,'^ 

S  =  cose^C()i'- 

cos^,' 

-Pr 

sin  f/- sin 

cosfV 

V  '  y 

The  intersection  equation  of  the  spoke  velocity 
vectors  is  derived  by  differentiation  of  the  linear 
intersection  equation  of  the  two  spokes  given  in 
Equation  5.2-6: 


Equation  5.2-10 

R'-S,  -t- R'-S,  =Wf+R- d'j  +R •  d'j -V,. 
whereby 

R'=  Range  Rate  of  the  MIDS  spoke  measured  in  the 
coordinate  system  of  the  MIDS  platform 

R  =  Range  Rate  of  the  ownship  spoke  measured  in 
the  coordinate  system  of  the  MIDS  platform 

T,  =  velocity  of  the  MIDS  platform 
The  velocity  'T.of  the  MIDS  platform  expressed  in 
coordinate  form  is  described  by  the  absolute  value 
Vj'  and  the  course  angle  t/in  the  (N,E)-plane,  the 

vertical  velocity  component  V;,  /  is  unknown: 


,  ^ 

r  . 

A 

M,i 

yy-cosr, 

t 

E,i 

I'i  i 

— 

v,'-sinr,' 

v„, 

0,1  J 

J 

Further  it  must  be  observed  that  both  the  elevation 
£/and  the  elevation  velocity  p/  of  the  MIDS  spoke 
are  unknown. 

With  the  above  derivations,  the  intersection  point  is 
now  formally  defined  by  the  two  vector  equations 
Equation  5.2-7  and  Equation  5.2-10: 

R'-S,=Xj+R-d'j 
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R'-S,  +  R'  S,  =  Wj  +R-d'j+R- &j  -  V, 

5.2.5  Determination  of  the  Siant  Ranges  R  and  R' 

In  order  to  gain  the  slant  ranges  R  and  R' ,  the  first 
Equation  5.2-7  will  be  separated  in  two  equations,  one 
which  expresses  a  two-dimensional  vector  equation 
and  a  one-dimensional  equation  for  the  third 
component: 


Equation  5.2-1 1 

f 


7? '-COS  £' 


.."I  (O'  ^ 


J 


''NJ 
Xc  ■ 


COS  (iJ, 

sin  (Pi 

Equation  5.2-12 

-7?'-sin£,'  =  X,)  j  +  R-O' 


+  R 


NJ 

O'e  / 


DJ 


The  scalar  Equation  5.2-12  can  be  resolved  in  the 
following  form 

By  using  this  result  in  Equation  5.2-11,  this  equation 
can  be  written  as 


Equation  5.2-13 


^RHxo^R-O'ojf 


cos^ 

sin^' 


fy  \ 


NJ 


U-R- 


(O'  ^ 

'  ^  NJ 


O' 


NJ 


This  equation  is  now  mapped  onto  the  vector 

(cos^/,sin^,.'): 

Equation  5.2-14 

^R'^-(x„j  +  R-0’„,y  =a  +  h-R 


whereby 


a  =  (cos(p  ',sin<p  ') 
h  =  {cos(p  ',sm(p  ') 


■^N.j 
Xp  f 


fo 

O' 


N,j 


f:j 


\ 

j 


If  one  projects  on  the  other  hand  Equation  5.2-1 3  on 
to  the  vector  (O' j  ,0' ^  j),  then  this  projection 
becomes  the  following  form 


Equation  5.2-15 

^R^-(x,,+R-0\jf  +f  (O'  O'  ) 


whereby 

<^  =  (o'«,, 


'NJ 


'EJ 


y 


By  means  of  the  Equation  5.2-14  and  Equation  5.2-15 
the  square-root  can  now  be  eliminated: 

Equation  5.2-16 

a  +  b-R=-  +  —  ‘ip'\j^O'^Ej) 
b  b 

or 

Equation  5.2-17 

a-b-c 

0'\j+0'^Ej-b^ 

Remark: 

R  is  not  a  function  of  the  unknown  height  information 
x^  j'of  the  MIDS  platform  and  therefore  the  height  is 

not  required  for  the  calculation  of  the  slant  range  of 
the  intersection  point. 

By  using  Equation  5.2-17,  R  will  be  substituted  in 
Equation  5.2-14  and  one  yields  the  slant  range 
R'  from  the  MIDS  platform  to  the  intersection  point: 


Equation  5.2-18 

R  '=  ±^{a +b-Fy+  , + o  F  y 

whereby 

„  a-b-c 

~  0'\j  +  0'^Ej-b^ 

Remark: 

In  this  equation  the  slant  range  R'  is  a  function  of  the 
height  of  the  MIDS  platform! 

The  question  to  be  asked  now  is  "How  strong  is  the 
influence  of  the  unknown  height  x,)  j  on  to  the  slant 

range  R'V 


2  2 

If  one  assumes  that  (7  y  O  h,  and 


o'„  ^  0 


ab-c 


R'  R'^ 

then  this  relation  should  be  far  less  than  1(e.g.  <0.04). 


5.2.6  Determination  of  the  Range  Rate  R 

The  method  to  determine  the  range  rate  is  similar  to 
the  method  to  determine  the  range. 

From  Equation  5.2-10  one  obtains 


11-7 


Equation  5.2-19 

R'-cos  e' 


'^cos 


r. 


sin^/ 
f 


-I-  R'-cos  e  '-0)  /-j 


sin  <p 
cos 


-  R'-sm  e^'-p  /• 

With  the  identities 


cos  <p  I 
sin  (p  / 


+  ^- 


fo '  ^ 

^  NJ 

+  R- 

(o'  / 

o  , 

o  V , 

(  f-j  J 

i?'-cosf,'  =a  +  h-R 
-  i?'-sin  '  =  x,j  j  +  R  ■  O'ly  j 


applied  in  Equation  5.2-1 8  and  succeeding  projection 
onto  the  vector  (-  sin  ,  cos  ^/  )  delivers 

Equation  5.2-20 


alignment  of  the  intersection  points  to  the  point  in 
time  will  be  achieved  by  means  of  the  extrapolation 

of  the  range  R^  by  means  of  the  slant  range  rate  R^ : 

Equation  5.3-1 


And,  as  agreed  in  section  5.1,  the  tilde  for  the 
extrapolation  will  be  left  out.  The  slant  range  R'  of  the 
MIDS  platform  is  not  going  to  be  extrapolated. 

6  Analysis  of  the  Credibility 
Preliminary  Remark: 

In  the  following  R  will  be  replaced  by  R^j ,  R'  will  be 

replaced  by  R^j' ,  and  .^will  be  replaced  by  R^,  in 
order  to  indicate  the  dependency  of  these  quantities 
upon  the  selection  of  the  spokes  and  Oj . 


^  {a  +  b-R)a}:-R-f-d 


whereby 

d  =  (-sin  ^,',cos^,') 
e  =  (-sin^,',cos^/)- 
/  =  (-sin^,',cos(3,') 


j 

('O'*,/ 

( \ 

^  NJ 

O'nj 


Again,  it  shall  be  mentioned  here  that  the  result  is  not 
dependent  from  V  /and  X  /. 


In  summary  it  can  be  said  that  it  is  really  possible  to 
determine  the  slant  range  R  and  slant  range  rate 

R  from  a  supposed  target  relative  to  the  ownship 
position.  Further  it  could  be  shown  that  a  slant  range 
R'  for  the  supposed  target  can  be  calculated  relative 
to  the  MIDS  platform,  which  however  is  dependent 

from  the  unknown  parameter  x  , ' . 

5.3  Alignment 

The  result  yielded  in  section  5.2  for  the  intersection 
points  of  the  spokes  must  be  time-aligned  to  the  point 
in  time  of  the  occurrence  of  the  ownship  spoke. 
Assuming  the  Radar  providing  the  spokes  has 
extrapolated  all  its  spokes  to  a  common  reference 
point  of  time  prior  to  delivery  to  a  mission  computer 
performing  these  triangulation  calculations,  then  all 
f^are  identically  equal.  However  the  intersection 

points  are  calculated  to  the  points  in  time  f/.  The 


The  calculated  range  and  range  rate  in  section  5.2 
represent  a  complete  three-dimensional  data  record 
of  the  Intersection  point  with  regard  to  location  in 
space  and  velocity.  This  data  record  of  an  intersection 
point  shall  now  be  investigated  towards  properties  of 
a  real  target. 

To  this  purpose  a  credibility  concept  will  be 
developed,  which  uses  concepts  of  the  Fuzzy  Set 
Theory.  In  this  application,  the  Fuzzy  Set  shall  be 
defined  as  the  set  of  realistic  target  behavior.  The 
credibility  parameter  evaluates  how  strong  this 
supposed  target  belongs  to  the  Fuzzy  set. 

From  one  or  many  parameters,  which  describe  the 
adhered  to  intersection  point,  characteristic  quantities 
are  calculated,  such  as  for  velocity,  altitude  and 
altitude  rate.  Then,  these  quantities  are  investigated 
towards  adherence  to  a  realistic  target.  If  this  is  the 
case,  then  a  credibility  measure  of  1  will  be  assigned 
to  this  parameter. 

If  one  can  exclude  with  a  high  probability  that  a  real 
target  behaves  like  the  intersection  point  for  one 
parameter,  then  the  credibility  measure  0  will  be 
assigned  to  this  parameter.  In  limit  areas  between 
realistic  and  unrealistic  behavior  of  the  intersection 
point  a  continuous  function  with  values  between  0 
and  1  is  going  to  be  applied. 

By  testing  as  many  as  possible  parameters  on  their 
credibility,  and  by  forming  the  product  of  the 
credibilities  of  the  individual  parameters,  it  is  possible 
to  separate  between  intersection  points  with  a 
credible,  realistic  physical  behavior  from  those  with  a 
less  credible  realistic  behavior,  or  even  from  those 
with  a  total  unreliable  behavior.  This  leads  to  a  pre¬ 
selection  of  MIDS  spokes,  which  may  belong  to  a 
dedicated  ownship  spoke  and  which  at  ail  describe  a 
realistic  target. 
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6. 1  The  Credibility  of  the  MIPS  Range 

MIDS  spokes  may  contain  a  range  estimation,  which 
was  gained  by  Kinematic  Ranging.  Assuming  there  is 
no  range  estimation  available,  then  every  range  must 
be  assigned  the  credibility  1. 


Suppose  now,  there  is  a  range  estimation  r\ available 
and  suppose  that  is  the  measure  of  credibility  for 


the  calculated  range 

If  Ry'  <(i  then  =0 
else  Ar,. '  then 


else  if  r'-Rf  <2^r' then 


g; = 2 


r'-R' 

I  il 

Ar,' 


else  G/  =  0 
end  if 


6.2  Credibility  of  the  Visibility 

For  this  credibility  test,  it  will  be  assumed  that  the 
range  Ry  >  0  is  acceptable  and  the  origin  of  the 
coordinate  frame  is  in  the  center  of  the  earth. 

Under  these  assumptions,  the  ownship  position  Pq  is 

Po(0,0,/tj  +  Pg).  Further  the  intersection  point  of 
the  spokes  shall  be  in  the  direction 
5^  =  (cosfyA-sinfy 

Under  these  conditions  one  can  form  the  linear 
equation  from  the  ownship  to  the  spoke: 


Equation  6.2-1 


j?y.(w)  =  Po  +rn-Sj 


For  m  =  Ry  one  gets  the  position  of  the  intersection 

point.  The  question  now  to  be  put,  is:  When  cuts  this 
straight  line  the  surface  of  the  earth  globe? 

This  happens  exactly  then  and  only  then,  when  the 
following  condition  is  fulfilled: 

Equation  6.2-2 

KI=|Pr("0| 


Equation  6.2-3 

=-s\nej{hj  +  Pj+  P/  - 

=-smej{hj  +R,  )-^P/  -(P,  +hi  Y  coi  £j 

In  case  the  discriminant  is  less  zero,  then  one  can 
conclude  that  no  real  solutions  of  the  equations  exist. 
The  visibility  is  given  then  and  only  then,  when  the 
discriminant  of  the  square-root  is  less  zero,  i.e.  when 
the  following  condition  is  fulfilled: 

Equation  6.2-4 

P^,(l-COS£  ) 

hj  > - - 

^  cos  Ej 

The  demanding  of  an  intersection  point,  when  m  is 
positive,  is  also  excluded,  in  case  the  elevation  angle 
is  not  negative. 

Therefore  the  visibility  is  not  given  or  at  least  doubtful, 
in  case  is  negative  and 

Equation  6.2-5 

P,(l-COSfJ 

hj  < - 

COSEi 

Since  7^2  ^  777, ,  one  concludes  that  Sj  (TTfj  )is  the 
first  intersection  point  of  the  straight  line  with  the 
earth  globe.  Visibility  is  given  then  and  only  then, 

when  Ry  <  m2  ■ 

The  considerations  up  to  now  are  based  on  pure 
geometric  approaches.  But  one  has  to  assume  that 
the  elevation  will  have  a  statistical  error  and  therefore 
the  question  of  visibility  can  only  be  answered 
statistically. 

The  straight  line  from  the  ownship  to  the  intersection 
point  touches  the  surface  of  the  earth  exactly  for  one 
angle  ,  if  the  following  condition  is  satisfied 


Equation  6.2-6 


R„ 


R.+h. 


In  case  the  elevation  angle  € 


is  less  Efj ,  then  the 


intersection  point  is  geometrically  not  visible.  But  still 
there  exists  a  certain  probability  that  the  true 
elevation  angle  to  the  intersection  point  is  in  fact 

larger  than  . 


On  the  other  side  ]f  >  Eg  ,  then  the  intersection 
Calculation  provides  geometrically  visible,  but  yet  it  exists  a  certain 

probability  that  the  true  elevation  angle  to  the 
intersection  point  is  in  fact  smaller  than  f^and 
consequently  the  intersection  point  remains  invisible. 
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It  is  therefore  legible  to  demand  -  assuming  normally 
distributed  errors  of  the  elevation  angle  directed  to 
the  intersection  point  -  that  the  intersection  point  is  at 
least  in  95%  of  all  cases  not  visible,  in  order  to  be  in 
the  position  to  assert  that  the  intersection  point  is 
geometrically  invisible.  This  means  -  assuming  a 

variance  O ^  for  the  elevation  angle  -  that  first 
the  straight  line  to  the  smaller  elevation  angle 
—  +  l-O  j  ^  is  not  visible. 

From  these  considerations  a  test  can  be  developed  for 
the  intersection  point  credibility  : 

If  R,.  <  0  then  Gf''  =  0 
end  if 

If  Bj  >  Eg  -  2a j  then  =  I 
else 

m^=-smej  (ft,  +  hj )-  yjR^  -cos^  e  j{r^ +hj} 
If  Rfj  <l.l'ffJ2  then  G^^  =1 

ft. -l.lw, 

else  if  ft  <\2-m^  then  GJ  =1 - 

^  O.l-ffjj 

else  Gjj^  =  0 

end  if 

end  if 


6.3  The  Heifiht  Creditibilitv 

It  will  be  assumed  that  the  slant  range  and  the 

elevation  Ej  of  the  intersection  point  are  given  at  the 
time  tj .  Further  it  will  be  assumed  that  the  earth  is  a 
sphere. 


The  altitude  of  the  target  above  ground,  calculated  by 
means  of  the  ownship  position  relative  to  the  target,  is 

Equation  6.3-1 


R,^  -  COS"  E: 

hj  =  hj  -F  Ry  •  sin  ^^  +  ■ 


2ft. 


The  credibility  test  for  the  supposed  altitude  of  the 
target  can  then  be  formulated  as  follows: 


If  Q  <  h^.  <  60,000  ft  then  G/  =  1 
else  if  60,000  ft  <  hj  <  80,000  ft  then 

g  H  ^  80  ,000  ft  -  h.j. 

20  ,000  ft 

else  if  -  5,000  ft  <  hj  <  0/f  then 
n  ^  5,000  ft  +  hj. 

^  5,000  ft 

else 

G/  =0 
end  if . 


6.4  Credibility  of  the  Vertical  Velocity 

It  will  be  assumed  that  the  slant  range  Rj  ,  the  slant 

range  rate  R^j ,  the  elevation  £j  and  the  elevation 
rate  Pj  of  the  intersection  point  are  given  at  the  time 
tj,  as  well  as  the  ownship  platform  vertical  velocity 
Vf,  .  Further  it  will  be  assumed  that  the  earth  is  a 

"j 

sphere.  Under  these  assumptions  the  vertical  velocity 
of  the  intersection  point  can  be  calculated: 


iin  £  ,  +  — - 

/  n 

J 


— —  ■  cos  ^  £ 
Re 

e 

■(  R. 

e  intersection  point 
( domains: 


+ 


J 

shall  be 


!!L< 

h-r  <  150 

m  . 
—  then 

S 

s 

600 

—  <  fij  < 

-  300 

g/  =  1 


-  then 


+  h  J 


m 

s 


)0  —  then 
s 


hr 


6.5  Maneuver  Test 

Altitude  and  altitude  velocity  determine  authoritatively 
the  maneuverability  of  a  flying  object.  Among  the 
altitude  and  altitude  velocity  exists  a  correlation, 
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according  to  this  a  low  flying  target  can  never  possess 
a  large,  negative  height  rate  and  a  high  flying  target 
should  not  have  a  large,  positive  height  rate. 

The  credibility  of  the  maneuver  of  a  flying  target 
should  recognize  this  inherent  correlation.  The  test  for 
this  credibility  should  make  transparent  the 
compatibility  of  the  height  and  height  velocity: 

If  3,300  m<hj  <  16  ,500  m  then  G,/  =  1 
else  if  hj.  <  3,300  m  then 

if  /ij.  >T,(h,)then  G/'  =/ 
else  if  hj  <T2(hj  )  then  =0 
else 

^  A/  _  ^^(hj. )  -  h^- 

~  T,(h,)-T,(h,) 

end  if 

else  if  hj  >  16 ,500  m  then 

if  hy  <  T*  ( hj  )  then  Gy^  =  1 
else  if  hj-  >7^  (hj  )then  Gy^^  =0 
else 

Q  A-/  _  T  2  ( h i-  )  —  hj- 

“  ~  T'2(h,)-T\(hr) 

end  if 

else  G,/^  =0 
end  if. 


The  quantities  T,  (hj.  )  {hj-  )  r*i  (Z;^  )  and  T\  Qij  ) 
represent  functions  of  the  threshold  values,  which  for 
a  given  height  hj^  provide  upper  and  lower  thresholds 


for  the  credibility  of  the  altitude  and  altitude  rate. 
These  functions  are  defined  as  follows; 


r,  (hj.  )=  -  300  m  Is 


hj.  -1-  2,000  m 

5,000  ni 


whereby  0  <  hj  <  3,300m  ; 


Tj  Qij  )=  -  600  m  /  s  ■ 


hj.  +  2,000  m 

5,000  m 


6.6  Credibility  of  the  Velocity 

In  the  following  it  will  be  assumed  that  at  time  tj  a 

valid  slant  range  Ry,  slant  range  raieRy,  the 
ownship  spoke  parameters  elevation  ,  the  elevation 
rate/3^  ,  the  azimuth  angular  velocity  (0 j  and  the 
ownship  platform  velocity  Vj  .are  given.  Further  it  will 

be  assumed  that  the  earth  is  a  sphere.  Under  these 
assumptions  the  absolute  velocity  of  the  intersection 
point  can  be  calculated; 


Equation  6.6-1 

Vr  =  ^tj  Oj  +  Ry  -  cos  E  j  -(Oj- 


-  sin  e y  ^ 
cos  (p  j 
0 


-  ■  Pi 


^sin  e y  ■  cos  ^ y  ^ 
sin  Ey  •  sin  <p y 
cos  E  j 


Further  it  shall  be  V-,.  =  |vj  | . 

The  test  for  the  credibility  of  the  velocity  V;.  of  the 
target  can  be  formulated  as  suggested: 

If  200  m  /  s  <  Vj.  <  700  m  /  s  then  Gy'’  =  1 
else  if  100  m  !  s  <  V  j  <  200  mis  then 
Q  V  _  Vy  -  100  m  / s 
''  ”  100  m  /  s 

else  if  700  m  /  s  <  Vj.  <  900  mis  then 
Q  V  _  900  m  I  s  -  Vj. 

■'  “  200  m  / 

else  G  y'^  =  0 
end  if 

6.7  Credibility  of  the  Intersection  Point 

The  credibility  of  an  intersection  point  can  now  be 
calculated  by  formulating  the  product  from  the 
credibilities  of  the  individual  parameters: 


whereby  0<hj  <  3,300m ; 


r‘i  (h.j  )=  150  m  Is 


25 ,000  m  -  h  j 


8,000  m 


whereby  hj  >16,500/w; 


T'i  {hj  )=  300  m  / s 


25,000  m  -  hj 
8,000  m 


whereby  hj  >16,500/«; 


Equation  6.7-1 


R  /'s  S  ^  H  /--t  H  jT-y  M  ^  V 

-^i/ 


The  elements  Gy  can  be  looked  at  as  elements  of  a 

matrix  G  .  The  matrix  G  will  be  noted  as  credibility 
matrix  in  the  forthcoming  chapters. 


if  one  defines  now  a  constant  threshold  value 
for  the  minimum  credibility  for  the  acceptance  of  an 
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intersection  point  as  a  supposed  target,  then  one  can 
define  an  acceptance  matrix  A ,  whose  elements  will 
receive  the  values  0  or  1 .  I.e.,  does  an  element  Aj 

receives  the  value  1,  then  this  intersection  point  under 
investigation  will  be  regarded  as  a  potential  target: 

If  G,  >  G,,„  then  A^  =  1 
else  A^j  =0 
end  if . 

The  matrix  A  defines  now  all  intersection  points, 
which  are  regarded  as  potential  targets.  In  the  second 
phase  of  the  analysis  the  remaining  targets  and 
supposed  targets  must  be  filtered. 

7  Association 

7.1  Alternative  1 

Association  is  the  identification  of  one  or  many  MIDS 
spokes,  which  define,  together  with  one  ownship 
spoke,  one  real  target.  The  aim  is  that  from  the 
analysis  of  the  density  of  the  intersection  points  -  by 
means  of  the  slant  range  and  slant  range  rate  -,  to 
achieve  the  association  of  the  MIDS  spokes  with  one 
of  the  ownship  platform  sensors  spokes.  The  analysis 
shall  take  place  in  three  individual  steps: 

1 .  Separation  of  good  and/or  believable  data  from 
not  accurate  and/or  unbelievable  data. 

2.  Selection  of  possible  slant  range  candidates  for 
each  ownship  spoke 

3.  Minimization  of  the  assignment  conflicts 

7.1.1  Separation  of  the  Data 

The  following  NxM-matrices  are  used: 

A  the  acceptance  matrix 
Rq  the  range  matrix  with  'good'  quality 
Rq  the  range  rate  matrix  with  'good'  quality 
R  the  slant  range  matrix 

R  the  slant  range  rate  matrix 
The  separation  of  good  and/or  believable  data  takes 
place  by  setting  a  threshold  with  respect  to  the 

required  minimum  quaiity.  The  minimum  quality  can 
thus  be  defined  individually  for  each  ownship  sensor, 
if  one  knows  significant  quality  differences  of  the 
utilized  onboard  sensors.  To  that  purpose  the 
following  conventions  shall  be  applied: 

The  acceptance  matrix  defines,  which  of  the  inputs  to 
the  other  matrices  are  valid.  One  can  eliminate  this 
matrix:  For  all  not  accepted  intersection  points  with 
Aj,  =  0  set  R,  =  oo  and  /?,,  =  0 .  After  this 
transformation  there  will  remain  for  the  further 
analysis  only  the  matrices  R,  R,  R  and  Rq  .  With 


the  above  explanations  and  agreements  one  can 
formulate  the  following  separation  algorithm  -  from 
now  on  called  '0-Cut'  -  which  eliminates  the  low 
quality  intersection  points  from  the  Slant  Range 

Matrix  R  and  the  Slant  Range  Rate  Matrix  R  ,  which 
do  not  exceed  a  certain  quality  minimum: 

Q-Cut: 

for  j  in  1  ..M  loop 
for  i  ini..  N  loop 
if  Gy<Qy  then 

else 

end  if 
end  loop 
end  loop 

It  must  be  pointed  out  that  the  Q-cut  cannot  be 
interpreted  in  that  way  that  the  intersection  points  are 
wrong  or  non-physical.  One  can  only  say  that  these 
points  are  to  a  great  extent  casual,  i.e.  one  could 
interpret  them  as  "background  noise"  of  intersection 
points  on  the  ownship  spoke. 

After  the  execution  of  the  Q-Cut,  the  decision  has 
been  taken,  which  elements  in  the  R  and  R 
matrices  will  be  trusted  and  those,  which  are  regarded 
as  casual.  For  the  further  proceeding  only  the 

matrices  R  and  Rq  are  still  required. 

7. 1 .2  Slant  Range  Candidates 

One  analyses  now  all  slant  ranges  and  slant  range 

rates  of  the  matrices  R  and  Rq  of  the  intersection 

points  of  the  MIDS  spokes  with  each  ownship 

spoke  Oj  with  regard  to  the  density  in  the  histogram. 

To  that  purpose  a  reasonably  defined  range  domain 
will  be  defined  over  which  a  slant  range  window  will 
be  moved  and  the  number  of  intersection  points  in  the 
window  will  be  counted. 

Let  be 

f,=\n-d-d,n-d-\-d)\  \<n<2A 

J ^  =  \^m  •  d  -  d,  m  •  d  -ir  d\  1  <  /«  <  24 
fs  ={24 -d -d,K,)-,  =[24-c/,oo); 

J_^=i^m-d-d—m-d-\-d\  l<m<24 
J_,,={-c^-24-d] 
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whereby  d  and  d  represents  the  half  width  of  the 
movable  window  and  may  have  for  example  the  value 

d  =  20km  and  d  =  50m  /  s  . 

With  these  definitions  one  is  now  in  the  position  to 
form  the  product  sets 

h  =ix,y)\xel„  A  yeJj, 

whereby  the  pairs  (x,>')  are  ordered  in  the 
mathematical  sense.  These  rectangles  form  a  partition 
of  the  set  of  permissible  slant  ranges  and  slant  range 
rates.  Further  it  is 

[jj„  =  [<5,  «>)= :  I,  \Jj„,  =  °°)=:  J 

n=l 


nt=-25 

ntM) 


/  X ./  =  [o,  oo)x  (-  oo,  oo) 

For  each  «e  {l,.--,25}and  /we{— 25,...,25}— {O} 
a  set  will  be  defined: 

=  eJ,\ 

M  contains  the  set  of  all  MIDS  spokes  S  ,  which 
do  possess  an  intersection  point  with  ownship  spoke 
Oj  with  regard  to  the  slant  range  e  /„  and  slant 

range  rate  R^j  e  . 

For  each  ownship  spoke  O ^  one  looks  for  up  to  three 
rectangles 


XJ, 


so  that  each  of  these  rectangles  contains  at  least  one 
intersection  point  and  none  of  the  other  rectangles 
X  should  contain  more  intersection  points. 

The  pairs  {R^j,R^j)  with  j  fix,  i  =  1,..., p, p  <  N 

are  the  potential  values  of  the  slant  range  and  slant 
range  rate  of  the  supposed  target.  The  corresponding 
sets  of  MIDS  spokes  causing  the  existence  of  the 

pairs  (Ry,Rjj)  in  the  windows  shall  be  denoted 

as 

L!j  —  set  of  MIDS  spokes  with  intersection  points  in 

K' 

=  set  of  MIDS  spokes  with  intersection  points  in 

Kj 

L^j  —  set  of  MIDS  spokes  with  intersection  points  in 
with  L) 


7. 1 .3  Minimization  of  the  Assignment  Confiicts 

An  assignment  conflict  occurs,  when  a  particular 
MIDS  spoke  S  has  more  than  one  intersection  point, 

i.e.,  with  at  least  two  Radar  spokes  O ^  and 

Oi(  ,ik^  j)  .These  intersection  points  are  contained 

in  different  sets  Kj  and  Kj^,  // =1,2,3,  k^j 
The  aim  is  to  minimize  the  assignment  conflicts,  i.e. 
exactly  one  rectangle  Kj  shall  be  assigned  to  each 

ownship  spoke  O  j  .  The  conflict  exists  then  and  only 
then,  when 

Equation  7.1-1 


M 


J=1 

vihereby  p  I  =  p ,  which  is  the  value  of  the  index 

p  of  the  adhered  to  window  . 

As  measure  for  the  size  of  the  assignment  conflict  one 
can  use  the  cardinality  of  the  set  of  the  multiple 
assigned  MIDS  spokes: 

M 

c= 

whereby  |f/|  denotes  the  cardinality  of  a  set  (7  . 

On  the  other  hand  an  assignment  is  desirable,  when  a 
maximum  D  of  intersection  points  can  be  used, 
which  calculates  according  to  the  formula 

M 


D 


. 3} 


The  optimal  assignment  for  the  hypothesis 
with  the  meaning 

a)  range  Ry  of  spoke  O ^  is  contained  in  the 
interval  \n"  -  d  —  d,n'’  -  d  +  d^  and 

b)  range  rate  Ry  of  spoke  Oj  is  contained  in  the 

interval  [w ”  -d  —  d, m"  ■  r/  +  z/ ] 
can  then  be  defined  by  demanding  that 


D-C  =  max ! 


In  this  case  a  maximum  of  used  intersection  points 
leads  to  a  minimum  of  assignment  conflicts. 

7. 1 .4  Assessment  of  the  method 

The  presented  method  requires  intense  calculations: 
Firstly  histograms  have  to  be  plotted  for  each  ownship 
platform  spoke  and  the  adhered  to  intersection 
points,  and  secondly,  an  optimization  process  has  to 
be  defined,  evaluating  the  many  hypotheses  from 
which  the  correct  assignment  can  be  suggested  with 
as  few  as  possible  assignment  conflicts. 
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7.2  Alternative  II 

The  aim  of  this  method  for  solving  the  assignment 
problem  is  to  investigate  the  usefulness  of  the  MIDS 
spokes.  This  method  requires  the  following  steps: 

1.  Separation  of  the  (almost)  unambiguous  spokes 
from  ambiguous  spokes 

2.  Allocation  of  MIDS  spokes  to  ownship  spokes. 

7.2. 1  Selection  of  the  MIDS  spokes 

The  separation  of  the  (almost)  unambiguous  MIDS 
spokes  from  the  ambiguous  MIDS  spokes  will  be 
achieved  by  setting  two  thresholds.  With  the 
definitions  from  7.1.1  and  the  threshold  A  to  limit 
the  maximum  number  of  valid  intersection  points  of  a 
MIDS  spoke  with  all  the  ownship  platform  sensors 
spokes,  one  can  now  formulate  a  combined  limiting 
process  called  ’  A  -  Q-Cut'. 

A  -  Q-  Cut 

for  j  in  1..M  loop 

k:=0 

for  iin  1..N  loop 

if  G^j  <Qy  or  k  >  A  then 

else 

k  :=k  +  1 

end  if 
end  loop 
end  loop 

The  A  -  Cut  guarantees  that  for  one  MIDS  spoke  5, 

not  more  than  A  intersection  points  -  with  at  least  the 
minimum  quality  Q  -  are  allowed  with  the  ownship 
spokes  Oj  ,  j  =  . 

A  further  effect  of  the  A  -  Cut  can  be  described  as 
follows:  If  there  are  too  many  intersection  points  of 
one  MIDS  spoke  with  many  ownship  Radar  spokes, 
then  one  can  argue  that  the  quality  of  the  information 
content  of  these  intersection  points  must  be  small 
and  can  therefore  be  ignored  in  the  assignment 
considerations. 

7.2.2  Assignment 

Having  performed  the  A  -  Q  -  Cut  in  section  7.2.1,  it 
may  be  the  situation  that  already  many  (or  most)  of 
the  intersection  points  are  rejected.  I.e.  for  the  further 
proceeding,  only  those  MIDS  spokes  have  survived, 
which  do  possess  a  sufficient  quality  and  do  not  have 
too  many  intersection  points  with  too  many  ownship 
spokes.  The  question  to  be  allowed  here  is,  whether 
an  ownship  spoke  at  all  has  still  intersection  points 
with  the  remaining  MIDS  spokes.  If  this  is  not  the 
case,  the  ownship  spoke  cannot  be  "deghosted". 


Assuming  now  that  some  MIDS  spokes  passed 
successfully  the  A  -  Q  -  Cut,  which  do  have 
sufficient  quality  and  do  not  have  too  many  common 
intersection  points  with  many  ownship  spokes.  Then 
in  the  final  step  one  tries  by  means  of  permutation 
logic  to  achieve  a  final  mapping  of  the  MIDS  spokes  to 
the  ownship  spoke. 

Chapter  8  shows  simulation  results  of  this  method. 

8  Simulations 

Simulations  for  the  Triangulation  algorithms  have  been 
performed  in  order 

>  to  test  the  formulas  for  the  intersection  points 
especially  the  coordinate  transformations 

>  to  fine-tune  the  credibility  tests  and  the  included 
fuzzy  logic  decisions 

>  to  unambiguously  filter  realistic  targets  out  of  a 
dense  random  scenario 

>  to  measure  the  processing  time 

The  programming  tool  used  was  MapleV  due  to  the 
excellent  graphical  possibilities.  The  algorithms  are 
written  in  a  special  programming  language  of  MapleV. 
For  the  generation  of  the  scenario  an  internal  random 
generator  was  used  which  generates  all  the  above 
mentioned  Spoke  data. 

The  results  have  been  displayed  in  a  two  dimensional 
polar  coordinate  system  (range  [km],  azimuth)  with 
the  ownship  at  the  origin  and  in  a  three  dimensional 

coordinate  system  ( [km],  [km],  altitude  [m]) 

with  the  ownship  position  at  (0,0, ).  All 

intermediate  results  (kinematic  data  of  each 
intersection  point,  each  single  credibility  test,  the 
whole  credibility  matrix  and  the  final  association 
results)  have  been  monitored  through  the  whole 
process,  so  that  the  behavior  of  the  algorithm  was 
traceable  and  possible  unreasonable  values  could  be 
recognized  immediately. 

8. 1  Generation  of  a  Random  scenario 

First  simulations  have  shown  that  triangulation  is 
most  accurate  when  the  Spokes  are  arranged  in  angle 
close  to  90°.  Taking  this  into  account,  we  can  set  up 
our  scenario  in  the  following  manner: 

>  Static  ownship  kinematic  data  (speed,  course, 
altitude.  Latitude,  Longitude) 

•  Generate  ownship  Spokes  (eventually  with  range 
estimate)  within  a  certain  azimuth  and  elevation 
range  (assume  a  typical  Radar  scan  volume  of 
±60°)  around  the  ownship  course;  azimuth  rate, 
and  elevation  rate,  generated  also  by  random  but 
with  reasonable  values  for  a  certain  range 

•  Generate  random  MIDS  positions 
(Latitude/Longitude)  and  kinematics  (speed  + 
course):  the  course  of  the  MIDS  platform  shall  be 
close  to  90°  of  the  ownship  course! 
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•  Generate  random  MIDS  Spokes  (azimuth  within 
±60°  around  the  MIDS  course,  azimuth  rate  with 
a  reasonable  value) 

•  Fix  configurable  fine  tuning  constants  for  each 
run 

•  Take  all  these  input  data  and  run  through  the 
algorithms 

To  generate  the  scenario  we  have  different 
possibilities  to  vary  the  environment: 

•  Number  of  Radar  Spokes  r 

•  Number  of  MIDS  Spokes  m 

•  Maximum  and  minimum  range  of  MIDS  platforms 

•  Maximum  and  minimum  velocities  for  the  random 
generator  (used  for  MIDS  platform  kinematics) 

•  Displayed  range 

To  fine-tune  the  algorithms  we  can  also  configure 
some  constants  used  in  the  credibility  tests  and  in  the 
association: 

•  Fuzzy  logic  limits  for  the  velocity  test  (4  values) 

•  Fuzzy  logic  limits  for  the  maneuver  test  (5  altitude 
and  4  altitude  rate  limits) 

•  Range  accuracy  for  both  range  credibility  tests 

•  Cut-off  possibility  for  the  credibility 

•  Association  thresholds  for  range  and  range  rate 

•  Assumption  for  MIDS  platform  altitude 

For  all  shown  simulations  following  symbols  are  used: 

+  =  incredible  intersection  points  (IPs) 

I  ■  cretiihie  iniorsncHoit  poinis: 

+  ■=  Ownship  position  (with  altitude) 

—  MIDS  platform  positions  3D  (without  altitude) 

0  =  MIDS  platform  positions  In  the  polar  display 

■“  “  Ownship  course  (length  «  velocity) 

—  =  MIDS  platform  courses  (length  «  velocity) 

— •  Radar  Spokes  (in  the  polar  display  — — ) 

..  MiDS-Snofn;-  (without  siovoticn  info! 

□  “  unambiguous  IP  (“=  real  target) 

For  the  tests  with  the  realistic  targets  the  following 
further  symbols  are  used: 

o—  =  Target  position  course 

All  simulations  have  been  performed  with  various 
input  data.  For  a  better  comparability  of  the  different 
tests,  same  input  data  (ownship  kinematic,..)  are  used 
for  the  figures  in  this  paper: 

Ownship  velocity:  300  m/s 

Ownship  course:  0°  (north  direction) 

Ownship  altitude:  5000  m 

Ownship  Latitude:  48° 

Ownship  Longitude:  1 1  ° 

Random  MIDS  velocities:  1 50  m/s..  500  m/ s 
Random  vertical  velocity:  -1 00  m/s..  1 00  m/s 
Maximum  distance  between  ownship  and  MIDS 
platforms:  100  km 


Before  we  start  with  the  test  results  some  words  to 
the  language  used  in  the  following  text: 

An  incredible  point  is  an  intersection  point,  which  has 
already  been  filtered  out  by  the  credibility  tests. 

A  credible  point  is  an  intersection  point,  which  has 
passed  successfully  the  credibility  tests. 

An  unambiguous  point  is  a  credible  point  which  has 
passed  successfully  the  association  decision  and  is 
therefore  determined  as  a  real  target  by  the  algorithm! 

The  first  test  which  was  set  up  was  a  complete 
random  scenario  with  6  Radar  Spokes  combined  with 
10  MIDS  Spokes  with  no  range  estimate  available  at 
all.  As  expected,  there  are  some  credible  and  even 
two  unambiguous  intersection  points  beneath  a  huge 
amount  of  incredible  points  (see  Figure  8.1-1  and 
Figure  8.1-2). 
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unambiguously  even  if  no  range  estimates  of  these 
two  Spokes  are  available! 


Figure  8.1-1:  6  Radar  +10  MIDS  Spokes  without 
any  range  estimates  available. 


Figure  8.1-2:  6  Radar  +10  MIDS  Spokes  without 
any  range  estimates  available  seen 
from  east. 


In  the  second  step,  we  examine,  how  the  algorithm 
behaves,  if  we  only  use  Radar  Spokes  with  ranee 
estimates  (possibly  kinematically  ranged  Radar 
tracks!).  In  Figure  8.1-3  and  Figure  8.1-4  you  can  see 
again  a  generated  random  scenario  but  with  an 
available  Radar  range  estimate.  In  this  case,  there  are 
only  incredible  points,  which  are  filtered  out  by  the 
very  strong  range  credibility  test! 

In  this  case  a  real  target  which  would  be  placed  in  the 
scenario  and  which  is  seen  from  2  different  origins 
(Ownship  +  one  MIDS  platform)  can  be  filtered  out 


Figure  8.1-3:  6  Radar  +  10  MIDS  Spokes  with 
range  estimate  for  the  Radar 
Spokes 


Figure  8.1-4:  6  Radar  +  10  MIDS  Spokes  with 
range  estimate  for  the  Radar 
Spokes  in  polar  coordinates 

The  same  result  is  achieved,  if  we  use  MIDS  Spokes 
with  and  Radar  Spokes  without  range  estimates  (see 
Figure  8.1-5). 
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Results: 


Figure  8.1-5:  6  Radar  +  10  MIDS  Spokes  with 
range  estimates  of  the  MIDS 
Spokes 

In  the  last  step  where  we  only  use  random  Spokes  we 
examine  the  behavior  of  our  algorithms  in  a  very 
dense  (20  Radar  +  30  MIDS  Spokes!)  scenario,  but  we 
assume  a  range  estimate  which  is  available  for  the 
Radar  Spokes.  Even  in  this  case  you  can’t  find 
unambiguous  points  anymore,  only  few  credible  ones. 
But  most  of  the  intersection  points  are  incredible  (see 
Figure  8.1-6). 


Figure  8.1-6:  20  Radar  +  30  MIDS  Spokes  with 
range  estimate  of  the  Radar  Spokes 
available 


•  Ambiguities  in  the  case  of  no  range  estimate 

•  Almost  all  intersection  points  are  filtered  out  in 
the  case  of  existing  range  estimates  (Ar=5000m) 


The  first  step  is  again  the  generation  of  a  realistic 
random  scenario  as  explained  above. 

In  the  second  step  one  realistic  target  [i.e.  it  behaves 
like  a  real  aircraft)  is  placed  somewhere  (input:  range, 
azimuth,  altitude,  course,  velocity  and  vertical 
velocity)  in  the  scenario  and  then  Spoke  data  for  this 
target  are  generated  “by  hand”  as  follows: 


>  Calculate  the  ownship  Spoke  data 

>  Place  one  MIDS  platform  in  the  scenario 

>  Calculate  the  MIDS  Spoke  from  this  platform  to 
the  target 

>  Simulate  with  and  without  range  estimates  of  the 
Spokes 

>  Place  a  second  (or  even  further  MIDS  platforms) 
in  the  scenario  and  generate  again  the  Spokes  to 
the  real  target 


The  ownship  data  are  constant  and  are  the  same  from 
above.  Additionally  the  kinematic  data  of  the  realistic 
target  is  variable,  but  for  comparison  reasons  in  this 
presentation  fixed  as  follows: 

Target  Range:  60  km 

Target  Azimuth  :  Jt/5 

Target  velocity:  400  m/s 

Target  vertical  velocity:  0  m/s 

Target  course:  -n/2  (west  direction) 

Target  altitude:  10000  m 

The  position  of  the  MIDS  platform  which  “sees”  the 
target  is  also  variable,  but  for  the  demonstration  here 
defined  as  follows: 

MIDS  platform  Range:  50km 

MIDS  platform  Azimuth:  n/2  (east  from  ownship) 

MIDS  platform  velocity:  400m/s 

MIDS  platform  course:  -n/A  (north-west  direction) 
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Figure  8.2-1:  6  Radar  and  10  MIDS  Spokes  with  a 
real  target  placed  in  the  random 
scenario 

Figure  8.2-1  shows,  that  in  the  case  of  existing  range 
estimates  (even  if  there  is  only  one  range  estimate 
available!)  the  real  target  is  filtered  out 
unambiguously.  The  range  accuracy  for  the  range 
credibility  tests  was  set  to  Ar=5000m. 

As  you  can  see  in  Figure  8.2-2  the  result  gets  even 
more  better,  if  you  assume  a  real  target  which  is  seen 
from  the  Ownship  and  from  two  different  MIDS 
platforms  with  range  estimates  available!  In  this  case 
the  snapshot  on  the  scenario  delivers  a  brilliant 
determination  of  the  real  target  even  if  you  have  a 
rather  dense  scenario  with  10  Radar  and  20  MIDS 
Spokes! 


Figure  8.2-2:  Real  target  seen  from  2  different 
MIDS  platforms. 

8.3  Processing  time  measurements 

The  simulations  give  us  also  the  possibility  to  measure 
the  total  processing  time  and  the  processing  times  of 
the  different  main  parts  of  the  triangulation  process 
(calculation,  credibility  tests  and  association). 

The  simulations  have  been  performed  on  a  PC  with  a 
Pentium  II  MMX  processor  (233MHz). 

The  processing  times  have  been  averaged  over  10 
runs. 

In  the  first  test  the  number  of  MIDS  Spokes  is  fixed  to 
10  and  the  number  of  Radar  Spokes  is  variable  from  1 
to  10;  Figure  8.3-1  shows  a  linear  increase  in  the 
processing  time. 


10  MIDS  Spokes 


Number  of  Radar  Spokes 

[-*-  Processing  Time] 


Figure  8.3-1:  total  processing  times  with  10  MIDS 
Spokes  and  variable  number  of 
Radar  Spokes 

In  a  second  test  the  number  of  Radar  Spokes  is 
constant  (6)  and  the  number  of  MIDS  Spokes  is 
variable.  The  result  is  shown  in  Figure  8.3-2.You  can 
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also  see  a  linear  increase  in  the  processing  time  with 
growing  MIDS  Spoke  number. 

In  a  third  test,  where  r-m  is  kept  constant  the 
processing  time  stays  also  nearly  constant. 

The  linear  time  behavior  should  change  in  that 
moment,  when  we  introduce  a  more  complicated 
association  algorithm,  e.g.  the  munkres  algorithm  with 
the  credibility  matrix  as  basis. 

The  processing  time  measurements  show  also  that  in 
the  current  design  it  is  partitioned  on  the  3  main 
processes  as  follows  (for  a  medium  number  of  Spokes 
6Radarx  10MIDS): 

•  Calculate  Intersection  points:  88% 

•  Credibility  tests:  10% 

•  Association:  2% 


6  Radar  Spokes 


j Processing  Time  ] 


Figure  8.3-2:  Processing  time  over  MIDS  Spoke 
number  by  maintaining  the  number 
of  Radar  Spokes  constant. 

8.4  Results  of  the  fine  tuning  constants 

All  the  above  mentioned  simulations  are  performed  for 
one  main  reason,  to  make  the  algorithms  as  accurate 
as  possible  by  ‘turning  the  buttons’  of  the  fine  tuning 
constants. 

The  achieved  best  results  of  these  parameters  are: 
Threshold  for  range  credibility  test:  5000m 
Cut  off  possibility  for  the  credibility  matrix:  95% 
Association  thresholds  for  range:  5000m 
Range  rate:  50m/s 
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Summary 

The  Bearings-only  Target  Motion  Analysis 
(BTMA)  problem  with  two  observers  on  ground 
consists  of  estimating  the  position  and  velocity 
of  maneuvering  targets  from  bearings-only 
measurements  corrupted  by  noise.  In  the  entire 
process,  there  are  two  parts  :  first  association  of 
the  measurements  on  each  observer  then  fusion 
of  tracks  made  by  each  observer  in  order  to  find 
the  trajectory  of  the  targets.  Fusion 
performances  are  highly  dependant  of  the 
trajectory  estimation  algorithm.  In  some  cases, 
conditions  of  interception  are  very  bad  :  few 
measurements,  targets  hidden  by  hill,  etc...  One 
way  to  improve  performances  is  to  introduce 
subjective  information  coming  from  user's 
expertise  or  external  processes.  We  present  here 
the  application  of  a  trajectory  estimation 
algorithm  based  on  the  Hidden  Markov  Models 
(HMM)  which  is  a  global  stochastic  method 
where  subjective  information  can  easily  be 
introduced. 


1.  Introduction 


The  Bearings-only  Target  Motion  Analysis 
(BTMA)  problem  with  two  observers  on  ground 
consists  of  estimating  the  position  and  velocity 
of  maneuvering  targets  from  bearings-only 
measurements  corrupted  by  noise. 


Figure  l.a. :  Bearings-only  Target  Motion 
Analysis  (BTMA)  with  two  observers 


We  have  an  observation  system  composed  of 
two  observers  on  ground  which  measure 
bearings  coming  from  maneuvering  targets.  The 
goal  is  first  to  associate  (on  each  observer)  the 
bearings  in  order  to  make  tracks,  each  track 
being  supposed  relative  to  a  single  target,  and 
then  to  fusion  tracks  made  by  each  observer,  in 
order  to  find  the  trajectory  of  targets.  The  track 
fusion  uses  trajectory  estimation  algorithms,  so 
if  the  estimation  has  a  good  quality,  the  fusion 
is  kept,  if  not,  the  fusion  is  rejected.  We  see  that 
algorithms  used  for  estimating  a  trajectory  have 
an  important  role  in  the  fusion  proeess. 

In  BTMA,  the  estimation  problem  is  non-linear 
and  sometimes  assumptions  made  in  algorithms 
as  Extended  Kalman  Filtering  can  be  too  rough. 
In  some  cases,  it's  often  usual  to  have  very  bad 
conditions  of  interception  :  few  measurements 
available,  or  none  during  a  few  periods  in  the 
case  of  ground  stations  used  at  a  low  altitude  in 
mountainous  areas.  In  these  cases,  one  way  to 
improve  the  trajectory  estimation  problem  is  to 
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use  subjective  information  given  by  the  user  or 
by  an  external  system. 

Many  ways  are  possible  to  introduce  subjective 
information  in  algorithms.  The  Hidden  Markov 
Models  (HMM),  which  is  a  global  stochastic 
method  adapted  to  the  non-linear  problems  and 
where  subjective  information  can  easily  been 
given,  is  used  in  this  paper. 

Let’s  see  the  problem  of  estimating  the 
trajectory  in  the  BTMA  with  two  observers, 
which  is  a  non-linear  problem  of  estimation. 

If  we  note  =  {X|,...Xj^.}  the  state  vector 
sequence  to  estimate,  which  are  position  and 
speed  in  BTMA,  and  =  {z^,...z^}  the 
observation  sequence  at  time  {t|,...t^.},  which  are 
bearings  in  BTMA,  then  the  goal  of  estimation 
is  to  determine  the  conditional  probability 
density  function  (pdf),  p(XyZ^.),  which  contains 
all  the  information  on  X.^. 

In  many  cases,  the  assumption  of  a  Gaussian 
pdf  is  done  which  needs  only  two  parameters 
(mean  and  variance)  to  be  completely 
determined. 

The  Kalman  filtering  is  applied  in  the  case  of 
linear  systems  with  Gaussian  initialisation.  The 
Gaussian  property  is  preserved  along  the  linear 
transformations  and  the  Bayesian  corrections. 
The  Kalman  filtering  gives  an  estimation  of  the 
mean  and  the  variance  of  the  state  given  the 
observations,  which  completely  determined  the 
Gaussian  pdf.  A  classical  approach  to  extend 
the  Kalman  filtering  to  the  non-linear  case  is  to 
expand  the  equations  in  Taylor  series  about  the 
nominal  values.  This  lead  to  the  Extended 
Kalman  Filter  (EKF).  This  extension  is  an 
approximation  and  the  performances  can  be 
poor  in  the  case  of  high  non-linearity  or  non- 
Gaussian  problems. 

An  other  point  that  is  important  to  see  is  that  the 
problem  of  estimating  a  pdf  is  compatible  with 
the  evolution  of  many  sensors  for  example  in 
the  BTMA.  Indeed  more  than  more  parameters 
are  measured  (bearings,  time  difference  of 
arrival,  Doppler,...)  and  this  measured  are 
translated  in  pdf  on  the  digitalized  state  space, 
not  necessarily  in  a  Gaussian  way  because  the 
behaviour  of  the  sensors  is  more  than  more 


well-known  and  this  knowledge  as  to  be  taken 
in  account  in  order  to  enhance  the  tracking.  On 
the  other  hand,  the  estimation  of  a  pdf  is 
multimodal  (i.e.  multitargets)  when  the  Kalman 
filtering  is  monomodal. 

For  the  tracking,  we  need  to  follow  the 
evolution  of  the  pdf.  In  this  paper  we  use  the 
Hidden  Markov  Model  (HMM).  In  a  general 
point  of  view  for  this  kind  of  algorithm,  we 
know  the  transition  law  p(x^/X|, ,)  and  the 
observation  law  p(z^/Xj,).  Our  aim  is  to  estimate 
p(x/z^)  recursively  in  time.  At  time  k-1,  we 
suppose  to  know  p(X|^  /z,^ ,).  Due  to  the  Markov 
assumption  for  x,  we  can  compute  the 
prediction  : 

P(xk  /  zk-i  )  =  j P(xk  !  Xk-1  )-P(xk-I  !  Zk-I  )  ■ 


Then  the  measurement  z^  is  used  to  do  a 
correction  in  order  to  obtain  p(x^/Zj,)  with 
Baye’s  formula : 


(Zk  /XK).g(XK  /Zk-i) 

P(zk  ^ 


Correction 


Prediction 


The  knowledge  of  p(x/Z|^)  gives  the  possibility 
to  give  an  estimate  x  I(x^)  of  the  state.  For 
example : 

I(Xk)=  jxKP(XK/ZK)Or 

XK 

I(xJ  =maxx^(p(x,/z,)). 


The  aim  of  the  methods  is  to  propagate  the  pdf 
and  estimate  I(x^). 


The  term  p(xjx^_i),  which  is  a  probability  of 
transition  allows  the  user  to  give  easily 
subjective  information  on  the  state  vector  and  so 
to  improve  the  performances  of  estimation  and 
then  the  performances  of  the  fusion.  This 
information  is  given  by  the  user  with  an  easy-to- 
use  interface. 
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2.  The  Interacting  Multiple  Models 

The  Interacting  Multiple  Models  (IMM)  in  the 
case  of  two  passive-only  sensors  has  been 
developed  in  [1].  This  algorithm  is  based  on  M 
Extended  Kalman  Filter  (EKF)  in  parallels, 
each  of  them  corresponding  to  a  particular 
maneuvering  hypothesis.  Knowledge  is 
introduced  by  modelling  the  change  of 
hypothesis  with  a  Markov  chain. 

3.  The  Hidden  Markov  Models 

Elements  on  Hidden  Markov  Models  (HMM) 
can  be  found  in  [2].  Applications  of  the  HMM 
to  the  problem  of  tracking  are  developed  in  [3], 
[4]  and  [5]. 

In  the  HMM  with  the  Viterbi  algorithm,  the  pdf 
is  estimated  on  a  fixed  grid  of  the  state  vector  x 
and  the  recursion  is  built  following  the  equation 
6(xJ  =  p(z^/xj.max^^  ^(p(x^/x^.,).5(x^J). 


One  particularity  of  the  HMM  is  the  fact  that  it 
doesn’t  introduce  a  formal  model  of  evolution  of 
the  target  as  in  the  IMM.  A  simple  hypothesis 
of  a  maximal  acceleration  for  example  is 
sufficient.  On  the  other  hand,  if  the  user  wants 
it,  it’s  also  possible  and  easy  to  introduce 
complex  prior  information  on  the  motion  of  the 
target. 

We  give  here  the  mains  equations  : 

Notations  : 

N  the  number  of  states  in  the  model, 

O, :  Observation  at  time  t, 

B  ;  observation  probability  distribution  in  the 
state  j 


B  =  {b/O,) },  b.(0,)  =  pr[0/state  j  at  t], 

l<j<N 

A  :  state  transition  probability  distribution 
A  =  {a^},  a-  =  pr[  state  j  at  t+1 /state  i  at  t], 
l<i,j<N" 

TZ :  initial  state  distribution 
71=  {ttJ,  7t,  =  pr[state  i],  l<j<N 
T  :  number  of  observations. 

Viterbi  Algorithm: 

Initialization 

6,(i)  =  7i,F(0,),  l<i<N 

Recursion 

5t(j)-max,<i<N[5t_i(i)aij]bj(Ot) 

'Pt(j)=argmax|<i<N[5,_,(i)aij] 

2<t^,  l<j<N 

Termination 

qj  =  argmaX|<i<N[8j(i)] 

We  get  the  state  from  the  backtracking  : 

qt  =  Vi(qt+i)-t  =  T-l,T-2,  ...1 

B  is  given  by  the  model  of  sensor  and  can  be  on 
analytic  form  (uniform,  Gaussian,...)  or  can  be 
given  in  a  numeric  form,  more  or  less  complex, 
in  the  state  space.  In  the  example  of  section  5, 
we  have  taken  a  Gaussian  distribution. 

A  can  be  fixed  with  many  criterion.  In  our  case, 
we  suppose  that  at  time  t  the  target  can  have  a 
maximal  acceleration  (for  example  ±3g).  This 
give  the  states  possible  at  time  t+1,  each  of 
them  having  the  same  probability.  As  we  see, 
this  hypothesis  is  very  simple  and  could  be 
easily  improved  in  order  to  enhance  the 
performances. 

In  our  case,  the  state  space  is  in  4  dimensions 
and  the  targets  are  highly  maneuvering.  The 
hypothesis  chosen  for  A  are  very  simple  and  the 
number  of  states  possible  can  be  very  high.  In 
order  to  reduce  the  computation,  we  have 
chosen  a  sub-optimal  version  of  the  HMM 
which  consists  on  keeping  the  states  the 
more  probable  at  time  t. 

If  we  know  nothing  about  the  target,  except 
some  large  bounds  on  position  and  speed,  7t  can 
be  an  uniform  distribution  on  the  state  space. 

On  the  other  hand,  it's  easy  to  introduce  prior 
information  to  reduce  computation. 
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4.  SUBJECTIVE  INFORMATION  GIVEN  BY  USER 

In  a  real  situation,  one  could  imagine  that  a 
system  made  of  two  ground  stations  is  installed 
in  an  area  where  enemy  planes  are  likely  to  fly 
or  even  to  attack. 


The  user  has  a  terminal  where  he  can  see  a  map 
of  the  area  concerned  and  the  position  of 
observers  (Figure  4.a). 


Figure  4.a.  :  Area  concerned  and  observers ’s 
position 

The  user  has  certainly  an  expertise  of  this  kind 
of  situation  and  he  might  also  have  a  few  tools 
and  external  information  such  as  maps, 
geodesic  information,  kind  of  targets,  and  so  on. 
In  our  example,  the  user  is  able  to  define  air 
corridors  where  he  thinks  a  plane  can  be.  So  the 
user  draws  a  few  points  on  the  map,  describing 
corridors.  Then,  giving  a  grid  size,  it  will  finally 
give  a  grid  2D  of  possible  positions  (Figure 
4.b). 


Figure  4.b. :  Grid  2D  of  possible  positions 


5.  SIMULATION 

We  take  the  same  subjective  information  as  in 
part  4. 

The  scenario  is  shown  on  Figure  5. a.  A  plane  is 
coming  first  in  a  straight  line,  then  it  turns  right, 
a  straight  line  again  and  finally  it  turns  left 
going  in  the  direction  of  one  of  the  observers. 


At  this  point,  the  user  can  also  give  more  Figure  5.a. :  Scenario 

information  on  parts  of  the  corridors  which  are 

more  likely  than  others,  or  decide  that  the  Each  observer  has  a  measurement  period  of  10 

distribution  on  the  grid  is  uniform.  seconds,  and  a  probability  of  detection  of  0,5. 

The  standard  deviation  is  3  degrees.  One  of  the 
observer  has  no  measurement  due  to  an  hill 
which  hides  the  plane  during  80  seconds. 
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Figure  5.b. :  bearings  measured  by  one 
observer 

We  see  that  the  conditions  of  interception  are 
not  very  good  :  few  measurements,  plane 
hidden  by  hill. 

In  order  to  compare  two  opposite  situations,  we 
have  applied  first  the  extended  Kalman  filter 
IMM  with  no  subjective  information  and  then 
the  HMM  with  the  grid  given  on  Figure  4.b. 
Both  algorithms  have  been  initialized  with  a 
batch  Least  Mean  Square  algorithm  with 
straight  line  hypothesis  on  the  first  part  of  the 
trajectory. 

The  position  estimated  by  the  algorithms  are 
shown  in  figures  5.c  and  5.d.  The  IMM  has 
some  erratic  behaviours,  particularly  on  the  end 
of  the  trajectory,  while  in  the  same  time,  the 
result  of  the  HMM  is  quite  good. 

One  can  also  see  the  distance  error  curve  on 
Figure  5.e.  The  HMM  curve  is  the  solid  line  and 
the  IMM’s  one  is  the  dashline.  After  the 
initialization  which  is  identical  for  both 
algorithm,  the  result  given  by  the  HMM  with 
subjective  information  is  very  good  compared 
to  the  IMM  one. 

Note  that  the  good  quality  of  the  result  is 
completely  dependant  of  the  subjective 
information  quality.  As  it  is  well-known  in  the 
theory  of  estimation  ;  when  the  subjective 
information  is  good  the  performance  of 
algorithms  increases  greatly,  but  when  this 
information  is  erroneous,  the  performance 
decreases  dramatically. 


O' - ' - ■ - ' - ^ - > 

0  5  10  16  20  25  30 

Measurement 

Figure  5.e. :  Distance  error  for  IMM  and  HMM 


Figure  5.c. :  Position  estimated  by  IMM  with 
no  subjective  information 


Figure  5.d. :  Position  estimated  by  HMM  with 
subjective  information 
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6.  CONCLUSION 

After  a  brief  review  of  the  Bearings-only  Target 
Motion  Analysis,  we  have  seen  one  way  to 
introduce  subjective  information  by  using  the 
Hidden  Markov  Models.  If  the  subjective 
information  has  a  good  quality,  the  result  of  the 
trajectory  estimation  becomes  very  good 
compared  to  a  classical  extended  Kalman 
filtering  without  prior  information.  This  can 
provide  a  solution  to  compensate  bad  conditions 
of  interceptions  as  seen  in  the  part  5  of  this 
paper. 

The  next  step  is  to  develop  an  automatic 
learning  process  which  uses  external 
information  and  human  knowledge  in  order  to 
propose  a  set  of  subjective  information  to  the 
user. 
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1.  INTEGRATION  OF  THE  HUMAN  OPERATOR  INTO  COMPLEX  AIR 
SYSTEMS  USING  ALTERNATIVE  CONTROL  TECHNOLOGIES 


Dr  G.M.Rood 

Systems  Integration  Department,  Air  Systems  Sector 
Defence  Evaluation  &  Research  Agency,  Farnborough 
Hampshire  GU14  OLX,  UK 


1.1  INTRODUCTION 

In  general,  a  combat  aircraft  can  be  described  as  manoeuverable 
airborne  weapons  platforms  which  contain  a  series  of  electronic 
and  other  systems  with  which  the  aircraft  is  controlled,  navigated, 
weapons  selected  etc.,  and  a  series  of  systems  which  provide 
protection  for  the  aircrew  throughout  the  performance  envelope 
of  the  aircraft  and  when  emergency  escape  is  unavoidable.  Most 
aircraft  platforms  have  an  operational  life  of  over  20  years  -  some 
a  lot  longer  -  and,  in  this  timescale,  although  the  basic  platform 
does  not  significantly  alter  -  mainly  for  cost  reasons  -  the 
avionics  and  crew  support  systems  fits  can  continue  to  advance  a 
number  of  generations  -  which  can  allow  the  airframe  to  retain  its 
operational  competitiveness  against  newer  designs 

The  speed  and  capacity  of  future  avionic  systems,  themselves 
increasing  in  complexity,  will  result  in  the  amount  of  information 
output  heavily  increased.  This  is  often  all  fed  to  a  single  pilot 
who  is  flying  the  aircraft  close  to  the  ground  at  around  450  knots 
or  more,  perhaps  in  bad  weather  at  night,  and  the  flying  process 
alone  needs  continuous  monitoring.  In  addition  s/he  needs  to 
keep  safe  control  of  the  aircraft,  find  the  target,  select  and  arm 
weapons,  be  aware  of,  and  react  to,  enemy  countermeasures, 
perform  complex  operations  with  smart  weapons,  etc.,  all  in  a 
degraded  environment  with  high  noise  levels,  high  vibration  and 
heat,  high  ‘g’  levels,  high  agility,  disorientation,  etc.  Out  of  this 
scenario,  one  of  the  primary  problems  is  the  amount  of  data  -  not 
necessarily  in  the  right  information  format  for  easy  digestion  - 
that  it  is  necessary  for  the  pilot  to  process  and  the  interaction  with 
the  displays  which  s/he  will  need  to  ensure  that  the  correct  inputs 
are  entered  at  the  right  time,  and  quickly  enough,  to  get  the 
operationally  relevant  information  out. 

Aircraft  avionic,  weapons,  navigation  and  other  systems  are 
unlikely  to  become  less  complex  and  this  increasing  complexity 
is  often  needed  to  counter  the  increasing  subtlety  of  enemy 
countermeasures,  there  is  a  tendency  to  need  more  inputs  to  a 
greater  number  of  systems  by  the  pilot  and  the  additional  time  to 
carry  out  these  extra  operations  is  not  generally  available. 

The  current,  and  traditional,  methods  of  data  input  or  selection  of 
systems  normally  require  the  use  of  the  hands  to  either  switch  a 
system  to  a  particular  state  or  enter  data  through  a  key-board. 
Most  current  aircraft,  both  civil  and  militaiy,  make  large  use  of 
keyboards  to  enter  a  wide  range  of  data  both  on  the  ground  and 
whilst  airborne.  Errors  do  occur  in  data  entry,  even  under  benign 
conditions,  and  sometimes  can  result  in  serious  consequences.  In 
military  aircraft,  data  entry  is  often  an  operational  requirement  in 
flight  and  experiments  have  shown  that  errors  of  around  2.2%  to 
2.9%  can  occur  in  high-speed  low-level  flight  [1-1]  and,  even  in 
the  office  environment,  typing  errors  in  the  region  of  1 .5%  occur, 
and  this  is  with  a  full  sized  keyboard  under  unstressed  conditions 
and  without  the  need  for  NBC  gloves  and  the  smaller  keyboards 


and  key  sizes  often  found  in  aircraft.  Next  generation  systems 
may  need  a  larger  number  of  data  inputs  and  to  increase  the 
manual  input  capability  of  the  pilot  either  requires  an  increase  in 
'typing'  speed,  a  larger  number  of  hands  or  an  alternative  control 
technique. 

In  civil  systems  errors  occur  traditionally  during  high  workload 
periods  [1.2]  -  often  during  a  runway  change  required  by  Air 
Traffic  during  approach  and,  for  the  military,  similar  errors 
could  be  expected  to  occur  in  aircraft  which  use  a  combination 
of  military  and  civil  systems  in  the  cockpit  (C-130J,  E3D,  C- 
17,  etc),  particularly,  perhaps,  in  the  more  demanding 
battlefield  support  role. 

More  demanding  operations  in  the  current  generations  of  fixed 
and  rotary  wing  aircraft,  particularly  at  night  and  in  poor  weather, 
have  increased  the  need  for  more  'eyes-ouf  operations,  which 
decreases  the  time  for  'head  down'  or  'head  in'  viewing  time,  both 
for  switching  operations  and  for  assimilation  of  information  from 
head  down  displays.  Similarly  the  speed  of  operations  has  led  to 
less  time  being  available  for  these  two  operations.  Progress  has 
been  made  towards  the  assimilation  of  visual  display  data 
through  the  move  towards  Helmet  Mounted  Displays  and  the 
time  reductions  in  switching  have  been  achieved  through 
ensuring  that  the  pilot  has  no  need  to  move  his  hands  from  the 
primary  aircraft  controls  during  high  workload  periods  by  the  use 
of  the  Hands  On  Throttle  And  Stick  (HOTAS)  concept.  Using 
Fitts  Law,  namely  that  the  time  to  move  the  hand  to  a  target  (in 
this  case  a  switch  or  button)  depends  only  upon  the  relative 
precision  required,  indicates  that  the  movement  time  -  a  summed 
combination  of  perceptual  processing,  cognitive  processing  and 
motor  processing  -  is  in  the  region  of  250  ms  (an  aircraft  moving 
at  500  knots  travels  in  the  region  of  80  metres  in  this  time).  Thus 
a  time  saving  of  around  250msec  is  achievable  by  minimising  the 
hand  movements.  This  generally  involves  the  provision  of  all  of 
the  necessary  manual  switches  on  either  the  throttle  top  or  the 
control  column  (stick)  top,  (HOTAS)  or  Hands  On  Collective 
And  Cyclic  (HOCAC)  -  for  helicopters  -  during  all  critical  flight 
operations.  An  example  of  HOTAS  controls  is  shown  in  Fig.  1 
for  the  AFTI  F-16  aircraft  [1-3.] 

As  the  capabilities  of  aircraft  will  continue  to  increase  through 
the  use  of  more  sophisticated,  and  a  wider  range  of,  sensors,  and 
control  through  software  increases,  the  ability  to  control  the 
aircraft  systems  will  inevitably  require  an  even  greater  number  of 
controls  -  many  of  these  being  necessary,  at  least  in  principle,  on 
the  HOTAS  controls,  as  many  are  time  critical  and  need  to  be 
operated  eyes-out.  The  rise  in  the  number  of  avionic  systems  and 
the  consequent  number  of  manual  switching  operations  necessary 
during  critical  phases  of  operations  (eg  beyond  FEBA  and  set-up 
&  attack  phase  of  a  ground  target)  has  resulted  in  a  gradual 
increase  in  the  numbers  of  switches/controls  per  crew  member  in 
the  cockpit  and  this  is  illustrated  in  Figure  1-2, 
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Figure  1-1  HOTAS  Controls  for  AFTI/F- 16  Aircraft 


The  increased  numbers  of  switches  and  controls  results  both  in 
longer  selection  and  switching  times  and  with  the  necessity  to 
look  head  down  into  the  cockpit  to  operate  the  correct  switch 
or  series  of  switches.  This  has  led  to  the  HOTAS  concept 
and,  on  HOTAS,  aircraft  of  the  1970’s  design  era  were  using 
around  16  stick  and  throttle  top  functions,  and,  whilst  some 
aircraft  designs  in  the  late  80’s  still  used  less  than  20 
functions,  some  fixed  wing  aircraft  were  up  to  33  functions 
and  helicopters  up  to  40.  Figure  1-3  illustrates  this  trend  and 
Table  1  shows  the  functions  allocated  to  HOTAS  for  a 
number  of  aircraft  [1-3]. 

There  are  some  indications  from  aircrew  that  the  numbers  of 
functions  are  becoming  both  difficult  to  remember  -  needing 
more  training  -  and  sometimes  difficult  to  operate  with  either 
standard  aircrew  gloves  or  NBC  gloves.  More  complex 
aircraft  systems  will  almost  inevitably  require  more  control 
mechanisms,  and  the  most  obvious  approach  is  to  increase  the 
number  of  HOTAS  keys  -  at  least  for  the  time  critical 
operations.  If  the  physical  space  is  no  longer  available  on  the 
throttle  or  stick,  the  temptation  will  be  to  use  ‘chording’  -  the 
simultaneous  use  of  two,  or  more,  (existing)  keys  to  select  or 
operate  systems  -  with  an  inevitable  increase  in  mental 
complexity. 

Where  the  numbers  have  reached  a  level  where  some  aircrew 
are  finding  some  difficulties  in  remembering  the  functions  of 


all  of  the  switches,  and  since  it  is  impracticable  to  label  the 


Figure  1-1  Number  of  controls / switches  per  crew  member 
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Figure  1-2  Trend  of  HOTAS  Switching 


switches  -  it  would,  in  any  case,  be  almost  impossible  to  either 
clearly  read  the  labels  in  their  position  in  the  aircraft  or  have  the 
time  to  read  the  labels  during  critical  parts  of  the  sortie,  -  there  is 
no  possibility  of  identifying  the  correct  switch  or  button  if  the 
memory  fails  or  falters  for  any  reason.  Since  a  large  percentage 
of  the  buttons/switches  are  for  critical  aircraft  functions,  and  thus 
will  be  time  critical,  any  delay  or  error  can  jeopardise  the  aircraft 
mission.  Further,  even  if  the  error  is  known,  the  procedures  to 
recover  from  such  errors  -  if  any  -  inevitably  take  time.  It  may 
not  always  be  clear  to  a  pilot  that  he  has  made  an  error,  or  that  he 
has  pressed  the  wrong  switch  or  button.  If  a  button  is  pressed 
and  the  expected  consequences  do  not  occur,  a  number  of  options 
appear  in  his  mind: 

•  The  switch  or  button  may  not  have  worked: 

-  Solution  ??  -  press  again  or  harder 

•  The  feedback  system  -  if  any  -  may  have  failed 

•  The  display  or  function  may  have  failed 

•  The  system  may  have  failed  -  is  there  any  feedback? 

•  It  may  be  the  wrong  button  -  which  one  now? 

All  of  these  take  time,  which  generally  is  in  critically  short 
supply  in  these  phases  of  flight.  A  well  implemented  alternative 
control  input  method  would  provide  alleviation  of  this  type  of 
operationally  critical  problem. 

A  potential  further  problem,  particularly  with  the  necessary 
physical  positioning  of  a  larger  number  of  switches  or  buttons  is 
the  difference  in  anthropometric  span  of  the  hand  &  fingers.  Not 
only  are  there  differences  in  the  populations  of  an  individual 
country,  but  there  are  statistical  and  practical  differences  between 
countries  -  sometimes  significant.  Currently,  a  number  of 
countries  are  accepting  female  aircrew  for  combat  aircraft,  and 
the  differences  in  HOTAS  systems  designed  for  male  aircrew 
may  elicit  problems  for  female  crew  with  differing  effective  digit 
length  and  hand-reach  anthropometry. 

Table  l-II  shows  an  example  of  the  differences  in  hand  length  of 
a  number  of  countries  and  of  a  number  of  trials.  The  average 
hand  length  for  males  is  191 .65  mm  with  an  average  spread  of  48 


Table  l.l-I  A  sample  of  functions  allocated  to  HOTAS 
controls 


Aircraft 

Design 

Date 

Throttle 

Functions 

Stick 

Functions 

Hand 

Controlle 

r 

Total 

F15C  Eagle 

1970 

11 

6 

0 

17 

F15E-front 

1982 

9 

6 

0 

15 

F15E-  rear 

1982 

0 

0 

6 

6 

Tornado  IDS  -  front 

1970 

4 

7 

4 

15 

Tornado  IDS  -  rear 

1970 

0 

0 

5 

5 

F-18AtoD  -front 

1975 

10 

8 

0 

18 

F-18E/F  -front 

1990 

10 

8 

0 

18+ 

F-18E/F 

1990 

0 

0 

6 

6 

AV8B+ 

1989 

9 

8 

0 

16 

Harrier  GR7 

1989 

17(8) 

17(8) 

0 

16+ 

Mirage  2000-5 

1987 

14 

9 

0 

23 

Rafale 

1988 

21 

11 

0 

33 

EF2000 

1991 

AMX 

1982 

6 

9 

0 

15 

F-16C/D  Falcon 

1983 

6 

8 

0 

14 

AFTI  F-16 

8 

10 

0 

18 

MIG-29 

7 

12 

19 

Tiger  -rear 

1985 

14 

12 

26 

-  front 

14 

12 

26 

AH  64  Longbow-rear 

1990 

6 

13 

0 

19 

-front 

0 

0 

11 

11 

EH101 

1984 

19(14) 

21  (12)  _ 

0 

40 

RAH66  Comanche 

1990 

14 

8 

0 

22 

MV22  Osprey 

1988 

9 

7 

0 

16 

A330  Airbus 

1990 

0 

3 

0 

mm.  Standard  deviations  are  in  the  region  of  9  mm,  which,  as  an 
estimate,  would  allow  a  HOTAS  mounted  set  of  switches  and 
buttons  to  be  designed  to  be  used  by  perhaps  some  70%  (>1  sd) 
of  the  pilot  population  without  undue  difficulty.  The  remaining 
30%  may  need  to  make  some  sliding  movements  around  the  stick 
or  throttle  to  accommodate  the  full  range.  The  female  average 
hand  length,  however,  is  an  average  of  176.3  mm  with  a  spread 
of  42.5  mm  and  an  sd  of  8.6  mm.  The  difference  in  mean  length 
is  some  16  mm,  which  could  provide  some  difficulty  in  design  of 
HOTAS  controls  which  must  be  operated  by  both  genders. 

Table  l-III  supports  this  hypothesis  with  figures  comparing,  in 
considerably  more  detail,  differences  between  UK  male  and 
female  hand  dimensions  [1-4].  As  an  indication  of  the  potential 
problems,  the  distance  from  the  ‘hand  crease’  -  representing,  in 
this  case,  the  apex  of  the  HOTAS  grip  -  to  the  finger  tips  displays 
an  average  difference  of  1.2  cm.  If  a  wider  range  of  male  and 
female  crews  need  to  be  accommodated,  then  this  difference  may 
be  increased  to  over  3  to  4  cm.  Similarly  for  span  between  the 
thumb  and  the  individual  digits,  which  gives  an  indication  of  the 
ability  to  operate  a  thumb  switch  and  another  with  one  of  the 
other  digits  average  differences  of  around  1 .3  cm  are  apparent. 
Similarly  the  true  digit  lengths  -  the  length  of  each  finger  -is 
shorter  for  females  by  around  5  mm  and  the  curved  hand  length 
is  shorter  in  females  by  some  1.65  cm.  Many  of  these  differences 
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Table  1.3-1  Hand  Length  Data 


Date 

Sample 

mm 

sd 

Range 

Spread 

Male 

UK  Military 

1982 

300 

191.30 

9,71 

169-224 

55 

Canadian  Military 

1974 

565 

191.90 

8.78 

170-212 

42 

German  Air  Force 

1966 

1006 

189.10 

8.70 

168-210 

40 

British  Army 

1970-75 

2000 

193.00 

10,30 

159-219 

60 

US  Army 

1970 

1482 

192,00 

8.70 

172-214 

42 

US  Army 

1966 

6682 

190.30 

9.60 

169-214 

45 

US  Air  Force 

1970 

148 

197.20 

9.30 

173-228 

55 

French  Army 

1973 

793 

189.00 

9,00 

174-205 

5th-95th% 

range 

UK  Civilian 

1981 

300 

191,00 

8.30 

165-219 

54 

mean  values 

(191.65) 

8.27 

(159-228) 

48 

Female 

UK  Military 

1982 

187 

176.10 

8,07 

159-197 

38 

US  Army 

1977 

1331 

174.40 

9.00 

155-196 

US  Air  Force 

1970 

211 

179.30 

8.60 

157-205 

43 

UK  Civilian 

1980 

92 

177.50 

10.10 

161-194 

5th-95th% 

range 

UK  Civilian 

1981 

200 

174.20 

7.20 

152-195 

43 

mean  values 

(176.30) 

(8,6) 

42.5 

may  be  able  to  be  accommodated  by  good  design,  but  there  must 
be  a  high  probability  that,  in  current  designs,  and  in  future 
designs  where  the  increasing  number  of  controls  surfaces  will 
perhaps  result  in  physically  smaller  switches  and  buttons,  the 
potential  competition  between  switch  numbers  and  available 
surface  area,  as  numbers  of  switches  or  tactile  controls  compete 
with  surface  area,  will  play  a  more  significant  limitation. 

1 .2  ALTERNATIVE  CONTROL  TECHNOLOGIES 

Many  of  these  problems  can  be  solved,  or  at  least  ameliorated 
in  operational  terms,  by  the  use  of  Alternative  Control 
Technologies.  There  are  five  fundamental  technologies  that  will 
allow  use  of  alternative  control  strategies.  These  are: 

*  Head-Based  Control 

*  Speech-Based  Control 

*  Eye-Based  Control 

*  Gesture-Based  Control 

*  Biopotential-Based  Control 

Of  these  five  alternative  control  systems,  head  tracking  and 
speech  control  are  considered  the  most  mature,  with  head 
tracking  control  already  being  in  operational  use.  Eye  tracking, 
whilst  technologically  advanced  and  useable  in  simulators  etc., 
still  requires  some  development  for  use  in  the  environment  of  the 
airborne  cockpit.  Gesture  based  and  biopotential  based  systems 
are  regarded  as  longer  term  solutions  for  the  airborne  cockpit, 
although,  of  course,  many  simple  EMG  systems  are  regularly  in 
use  as  aids  for  persons  with  disabilities. 

There  is  a  gradual  transition  to  use  of  some  of  the  alternative 
controls  and  this  is  apparent  in  the  next  generation  operational 
aircraft  of  the  Eurofighter,  F-22  and  Rafale  type.  Experimental 
flying  in  the  UK,  France  and  USA  are  using  Helmet  Mounted 


Table  l.l-II  Details  of  Hand  Dimensions 

Male  Female 


Mean 

Sd 

Range 

Mean 

Sd 

Range 

Finger  number  to  hand  crease 

Digit  2 

Left 

12,26 

0.81 

10.2-14.4 

11.16 

0.70 

9.4-13.4 

Right 

12.21 

0.79 

10.2-14.9 

11.17 

0.69 

9.3-13.1 

Digit  3 

Left 

13.51 

0.92 

11.1-16.0 

12.12 

0.79 

10.3-14,4 

Right 

13.40 

0.87 

11.3-16.5 

12.09 

0.78 

10.3-14.6 

Digit  4 

Left 

12.44 

0.95 

9.8-15.0 

11.00 

0.86 

9.0-13.5 

Right 

12.31 

0.90 

10.0-14.9 

10.97 

0.85 

9.2-13.6 

Thumb 

Left 

6.02 

0.50 

4.7-7.6 

5,50 

0.44 

4, 3-6,9 

Right 

6,11 

0.48 

4.9-7.8 

5.62 

0,43 

42-7.0 

Digit  5 

Left 

9.13 

0.98 

7.0-12.3 

8.28 

0.80 

6.2-10.9 

Right 

9.49 

0.90 

7.1-11.8 

8.31 

0.89 

6.3-11.1 

Sights  &  Displays  which  are  fully  dependent  upon  head  tracking 
to  provide  the  helmet  pointing  capability.  Some  production 
systems  are  in  service  use  in  a  number  of  countries  (Russia, 
Romania,  South  Africa,  Israel...)  and  are  generally  simple  sights, 
but,  even  at  this  stage  of  development,  allow  significant  increases 
in  operational  performance,  when  correctly  integrated  with  a 
suitable  weapon  system.  Similarly  Voice  Control  systems  are 
flying  experimentally  and  being  used  during  simulation  to 
demonstrate  significant  operational  benefits.  Eurofighter  will  use 
voice  control  as  an  integrated  part  of  avionics  control  and  Rafale 
will  have  the  use  as  an  option.  Eye  tracking  would  appear  to  have 
strong  potential  in  military  systems,  particularly  when  used  in 
conjunction  with  head  tracking.  Whilst  head  tracking  gives  a 
good  indication  of  the  position  in  which  the  head  is  pointing  - 
and  is  fully  adequate  for  a  number  of  applications  -  it  does  not,  of 
course,  necessarily  show  what  the  pilot  is  actually  looking  at  (ie 
where  the  gaze  is  fixed).  A  number  of  techniques  exist  to 
measure  eye  position,  some  more  robust  than  others  in  an 
operational  environment,  and  many  are  used  in  the  simulation 
environment,  where  many  practical  environmental  limitations  are 
minimised.  The  electrophysiological  measures,  and  measures  of 
gesture,  are  currently  limited  in  their  application  in  the 
operational  cockpit,  although  there  is  considerable  potential  for 
these  type  of  systems  in  the  2010  to  2015  timescales.  The 
Adaptive  Interface  between  the  pilot  and  aircraft  systems,  in 
which  the  aircraft  will  infer  the  state  of  the  pilot  and  the  pilot  and 
aircraft  will  have  a  knowledge  of  the  state  of  each  others  systems, 
will  need  the  capabilities  of  these  technologies. 

A  further  important  practical  issue  that  must  be  addressed  arises 
from  the  implications  that  the  alternative  control  technologies 
involved  with  head  pointing,  eye  tracking  and  voice  control  all 
require  some  sort  of  head  mounted  system.  In  the  cockpit 
environment  these  systems  are  generally  head  mounted  on  the 
flying  helmet  or  an  existing  helmet  mounted  display.  However, 
care  must  be  taken  to  minimise  both  the  total  head  mounted  mass 
and  the  centre-of-gravity  offsets  caused  by  the  additional  head 
mounted  equipment,  as  there  are  flight  safety  issues  involving 
injury  to  aircrew  from  such  systems,  which  can  have  serious 
implications  in  operational  use. 

1.3  HEAD  POINTING 
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Of  the  potential  alternative  control  technologies,  Head 
Movement  Tracking  is  undoubtably  the  most  used  and  is,  and  has 
been,  in  operational  use  in  a  number  of  forms,  for  a  number  of 
years,  with  a  number  of  airforces.  Such  tracking  technology  is  of 
primary  importance  as  it  is  fundamental  to  all  future  systems  that 
will  wish  to  use  Head  or  Helmet  Mounted  Displays  -  in  any 
conceivable  form  -  or  any  type  of  visually  coupled  system. 

Currently,  the  majority  of  aircraft  carrying  out  a  missile  attack  on 
a  ground  or  airborne  target  must  point  the  nose  of  the  aircraft 
towards  the  target  in  order  to  suitably  align  the  enemy  aircraft  on 
the  weapon  aiming  displays  on  the  HUD  to  lock-on  the  weapon 
prior  to  firing.  This  is  not  only  a  time  consuming  approach,  but 
may  require  the  aircraft  to  perform  tortuous  manoeuvres  in 
pursuit  of  the  also  manoeuvering  target  aircraft.  Figure  1.2-4 
illustrates  the  sustained  and  instantaneous  manoeuvre  capability 
that  is  currently  required  from  an  air-to-air  combat  fighter,  in  this 
case  the  FI 6. 

Unfortunately  the  human  body,  being  developed  over  a  few 
million  years  for  a  less  stressful  environment,  does  not  respond 
well  to  these  violent  manoeuvres  and  technologically  complex 
and  ingenious  methods  of  protecting  the  body  must  be  employed. 
Currently  airframe  soft  limits  in  the  region  of  9  'g'  are  in  use  in 
current  production  and  future  aircraft  and  the  protection  of  the 
crew  to  these  levels  is  complex  and  cumbersome. 

The  emergence  of  the  technology,  over  the  last  1 5  years,  to  allow 
flight  worthy  Helmet  Mounted  Displays  (HMD)  [1-5,  1-6,  1-7] 
and  the  development  of  accurate  flight  worthy  Head  Pointing 
Tracker  Systems  (HPS)  has  allowed  methods  other  than 
manually  boresighting  the  aircraft,  to  be  used  to  enhance  weapon 
delivery  techniques. 

Future-current  and  next  generation  weapon  systems,  particularly 
air-to-air  close  combat  engagements,  will  be  able  use  an 
alternative  form  of  control  system  that  will  integrate  the  HMD, 
the  HPS  and  the  missile  seeker  head.  Figure  1 .2-5. 

This  enables  the  missile  seeker  to  be  driven  by  the  head  pointing 
system  to  look  in  the  direction  that  the  pilot's  head  is  pointing, 
and,  as  the  pilot  sights  the  target  aircraft  in  his  helmet  mounted 
sight,  for  the  missile  to  lock-on  and  be  fired  at  high  off-boresight 
angles,  without  the  necessity  for  violent  manoeuvering  of  the 
aircraft. 

There  are  a  number  of  technologies  available  to  provide  head 
tracking;  mechanical,  inertial,  acoustical,  optical,  but  currently 
(1998)  the  most  mature  is  the  a.c.  magnetic  tracker.  This  system, 
in  its  well  developed  form,  provides  the  bulk  of  the  systems  in 
use  in  the  world.  It  provides  good  accuracy  when  mapped  into 
individual  cockpits,  and,  as  importantly,  it  provides  a  large 
headbox  in  the  cockpit  allowing  the  pilot  to  obtain  head  position 
data  from  essentially  the  whole  effective  cockpit  volume. 
Accuracy  is  variable  across  the  cockpit  volume,  particularly 
towards  the  extremities  of  the  headbox,  but  accuracies  in  the 
region  of  4  mR,  with  no  worse  than  8mR  are  achievable  (1998), 
and  degredation  of  the  sensor  signals  are  generally  regarded  as 
‘graceful’.  In  the  first  instance,  this  is  fully  acceptable  for  use  in 
close  combat  in  ‘one  v  one’  or  ‘one  v  more’  engagements  with 
short  range  missiles.  The  missile  seeker  head  is  slaved  to  the 
pilots  head  movement  and  the  Field  of  Regard  of  the  seeker  head 
is  large  enough  to  negate  the  need  for  very  high  accuracy  (< 
3mR).  Significant  savings  in  target  engagement  times  are 
achievable  with  this  simple  system.  For  air-to-ground  targeting, 
this  simple  system  also  has  some  advantages.  With  a  targeting 
FLIR,  the  FOV  is  generally  small,  typically  10  degrees,  and 
searching  for  a  ground  target  with  a  hand  controller  and  head 


down  display  is  difficult  and  time  consuming.  Designation  of  the 
ground  target  through  the  helmet  sight,  with  the  FLIR  slaved  to 
the  head  movement,  will  again  allow  significant  time  savings  to 
be  made  in  target  designation,  particularly  for  targets  of 
opportunity.  As  part  of  this  targeting  system,  it  would  be  feasible 
to  use  the  voice  command  as  a  control  medium,  -  an  alternative 
to  using  a  manual  action  -  to  mark  and  lock  onto  the  ground 
target.  Where  ‘eyes  out’  operations  are  at  a  premium  and  the 
manual  processes  require  glances  inside  the  cockpit,  voice 
command  would  provide  an  alternative  control  strategy. 

In  a  similar  way,  the  use  of  visually  coupled  systems,  to  allow  a 
pilot  to  fly  operationally  in  night  and  poor  weather  conditions  by 
the  use  of  sensor  derived  images,  place  an  essential  need  upon 
the  measurement  of  head  position.  In  helicopters  of  the  US 
Apache  type,  the  Pilot  Night  Vision  System  (PNVS)  couples  the 
nose  mounted  Infra-Red  sensor  directly  to  the  pilots’  eye-piece 
on  the  helmet  display,  and  slews  the  sensor  to  the  pilots  head 
position,  lining  up  the  sensor  point  of  regard  with  the  pilots’  eye, 
using  the  optical  head  tracking  system.  Similar  systems  exist  on  a 
number  of  other  helicopters,  either  experimentally  or  close  to 
operational  deployment.  In  current  operational  deployment,  the 
pilot  retains  a  view  of  the  outside  world  to  help  retain  some 
external  situational  awareness,  but  eventually  it  may  be  necessary 
to  operate  without  a  direct  external  view,  if  the  ultimate 
protection  against  optical  blinding  weapons  (e.g.  lasers  etc.)  and 
other  considerations  become  necessary.  This  situation  then, 
essentially,  becomes  the  beginning  of  the  Virtual  Cockpit  and 
some  of  the  limitations  of  the  systems  needed  to  enable 
implementation  of  this  ‘virtual  cockpit’  philosophy  become 
apparent.  One  of  the  major  limitations  is  in  the  temporal  delays 
(also  called  transport  delays  or  latencies)  that  are  induced  by  the 
system  processing  needed  between  the  sensor  inputs  from  the 
outside  world  and  the  sensor  output  to  the  crews  eye.  For 
instance,  the  delays  between  the  pilot  moving  his  head  to  look  at 
a  particular  point  in  space  and  the  sensor  fixing  on  that  same 
point  should  be  minimised,  and,  whilst  not  yet  well  defined  by 
flight  experimentation,  figures  of  20  to  50  ms  are  often  quoted  - 
the  shorter  the  better.  The  delays  depend  upon  the  level  of 
processing  needed  in  both  the  head  movement  and  the  image 
transfer  system,  and  the  more  complex  the  system,  generally  the 
longer  are  the  delays.  For  next  generation  helmet  mounted 
displays,  where,  for  instance,  flat  panel  displays  may  be  the 
image  source  on  the  helmet,  the  considerable  levels  of  processing 
needed  to  carry  out,  for  instance,  distortion  correction  in  off-axis 
optical  systems,  needs  good  and  careful  system  design  to 
minimise  transport  delays. 

The  operational  advantages  of  the  use  of  this  head  tracking 
technology  has  been  shown  in  flight  trials  both  in  the  USA, 
where  live  missiles  have  been  fired  at  drones  (BOXOFFICE)  and 
in  the  UK,  where  air-to-air  close  combats  have  been  carried  out 
in  1  V  1  trials  (JOBTAC)  significant  reductions  in  target 
acquisition  and  engagement  times  are  apparent. 

The  use  of  a  helmet  mounted  display  and  head  tracking  system  in 
an  FI 6,  combined  with  a  missile  capable  of  acquiring  targets  of 
over  60  degrees  off-boresight,  has  allowed,  in  live  firings  against 
a  QF  106  target  drone  at  0.7M,  successful  intercepts  at  57 
degrees  off-boresight  whilst  the  target  was  manoeuvering  at  5g. 
Similarly,  in  one-on-one  or  two-on-two  air-to-air  combat 
between  a  MIG-29’s  fitted  with  a  simple  Russian  helmet 
mounted  sight  and  using  a  AA-11  (Archer)  missile,  and  F16’s 
with  no  helmet  sight,  the  MIG-29  was  able  to  attain  the  major 
number  of  first  shot  missile  releases  by  use  of  the  Helmet  sight 
system.  To  pass  the  head  position  information  to  the  missile 
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Figure  1-3  Instantaneous  (upper)  and  sustained 
(lower)  manoeuvre  capability  of  the  F-16/  79 

seeker,  the  MIG-29  used  an  electro-optical  head  tracking  system. 
[1-8]. 

Similarly,  at  Famborough  in  the  UK,  trials  have  been  flown  of 
one-on-one  combat  in  a  Jaguar,  using  a  captive  AIM9L  and  a 
standard  Mk4  UK  flying  helmet  fitted  with  a  simple  DERA/GEC 
sight  providing  weapon  systems  information  through  an  LED 
display  and  an  AC  electro-magnetic  head  tracker.  Target 
acquisition  and  engagement  times  were  significantly  reduced, 
with  off-boresight  acquisitions  up  to  60  degrees  being  achieved. 

Head  Tracking  can  also  be  used  to  designate  ground  targets  from 
the  air,  or  to  point  narrow  FOV  sensor  systems  at  targets  -  and 
these  generally  replace  manual  control  systems  that  are  displayed 
on  a  HDD.  Hunting  for  a  target,  in  a  moving  aeroplane,  with  a 
narrow  FOV  sensor  (likened  to  looking  for  a  target  through  a 
straw)  can  be  difficult  in  the  best  of  conditions  and  may  take 
longer  than  is  acceptable.  By  the  use  of  either  a  Helmet  Sight 
with  Head  Tracking,  or  with  the  addition  of  Eye  Tracking,  this 
type  of  operationally  essential  process  can  be  considerably 
shortened  and  higher  accuracies  attained.  UK  trials  have  linked 
together  such  a  system  enabling  the  FLIR  sensor  in  TIALD 
(Thermal  Imager  and  Laser  Designator)  to  be  located  directly  on 
a  target  of  opportunity  using  a  helmet  sighted  system  in 
conjunction  with  the  head  tracker. 


As  with  most  systems,  however,  whilst  there  may  be  significant 
operational  shorter  term  advantages,  there  are  also  some  longer 
term  restrictions  in  the  systems  use  of  Helmet  Mounted  Head 
Pointing  Systems.  One  of  those  comes  from  the  inability  of  a 
correctly  strapped-in  pilot  to  move  his  head  much  more  than  90 
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Figure  1-4  Use  of  head  pointing  in  an  off-bore¬ 
sight  capability  weapon  system 

degrees  to  the  left  or  right.  Figure  1-6  shows  the  head  pointing 
envelope  of  a  pilot,  in  full  flying  clothing,  in  a  fast-jet  strike 
aircraft  cockpit,  and,  whilst  the  envelope  is  acceptable,  it  is 
limited  by  the  available  head  movement  of  the  human  body.  If 
however,  a  further  alternative  control  method,  in  the  form  of  eye¬ 
tracking  is  utilised,  then  the  useable  envelope  is  significantly 
increased.  This  will  allow,  on  average,  tracking  to  around  ±  140 
degrees  in  the  horizontal  plane,  compared  to  ±  90  degrees  for 
head  tracking  and  up  to  90°  in  the  vertical  planes,  compared  to 
55®  with  head  tracking  (in  an  aircraft  with  the  restricted  rearward 
and  upward  movement  of  the  head  from  an  Ejection  Seat 
Headbox). 

Thus,  it  should  be  technologically  possible  to  targets  in  the  rear 
hemisphere  -  or,  at  least  be  able  to  input  information  into  the 
weapon  systems  as  to  the  position  of  target  aircraft  outside  of  the 
conventional  radar  systems  field-of-regard  or  missile  seekers 
FOR  [  unless  missile  design  changes  ]  -  but  not,  perhaps,  outside 
of  next  generation  thermal  sensors  FOR.  The  Russian  Vympel 
Design  Bureau  is  reported  as  having  tested  a  rear  engagement 
capability  in  1993  on  a  Sukhoi  Su-27.  The  control  authority  of 
thrust  vectoring  allowing  a  rearward  shot  without  the  missile 
losing  control  as  it  initially  flies  backwards,  [1-8]. 

One  of  the  assumptions  with  head  tracking,  and  noted  peviously, 
is  that  the  head  position  reflects,  in  hopefully  an  accurate  way, 
where  the  eyes  are  looking.  Whilst  this  may  be  broadly  true, 
there  are,  of  course,  many  times  when  this  will  not  be  the  case. 
Measurement  of  eye  position  and  gaze  point,  along  with  head 
position,  will  nullify  the  errors  in  these  assumptions. 


1.4  EYE  TRACKING 

There  are  a  number  of  distinct  advantages  to  using  eye  position 
sensing  as  an  operational  tool.  A  primary  advantage  is  the 
increased  surface  area  that  can  be  swept  by  the  eye  in  comparison 
with  the  head  movement  range  in  the  cockpit  environment. 
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Figure  1 .6  shows  this  clearly,  with  the  average  horizontal  range 
increasing  in  the  order  of  50  degrees  and  the  vertical  range  by  35 
degrees.  This  increase  in  the  swept  spherical  surface  area  should 
provide  the  ability  to  locate  and  designate  a  wider  range  of 
targets  particularly  in  air  combat,  and  this  may  be  of  a  particular 
advantage  during  high  ‘g’  manouevres,  where  head  movement 
may  be  restricted,  but  eye  movement  not,  and  an  adequate  field 
for  targeting  available,  from  eye  movement  alone,  with  the  head 
being  necessarily  held  in  a  fixed  position  during  the  high  ‘g’ 
manoeuvering  part  of  flight. 

Experimental  trials  at  Famborough  (1998)  in  the  centrifuge  have 
shown  that,  at  least  up  to  8  ‘g^\  eye  fixation  and  change  of  eye 
fixation  on  a  range  of  targets  could  be  accomplished  with 
subjectively  acceptable  results.  The  equivalent  trials  using  head 
pointing  was  problematical  at  the  higher  ‘g’  levels.  Objective 
results  are  in  the  process  of  analysis. 

A  further  application,  in  air-to-ground  attack,  could  be 
improvements  in  the  accuracy  of  designation,  by  the  eye,  of 
ground  targets  for  smart  weapons,  the  accuracy  being  improved 
by  the  capability  of  the  eye  to  be  naturally  physiologically 
stabilised  in  the  presence  of  turbulence  or  the  low  frequency 
buffet  normally  present  in  the  ground  attack  phase  [1-9].  The 
accuracy  of  head  pointing  under  these  conditions  of  turbulence  is 
unlikely  to  be  of  adequate  quality  for  precision  designation. 


Eye  Tracking  has  also  some  similar  potential  outside  of  the 
current  conventional  fast  jet  or  helicopter  cockpit  or  cabin, 
particularly  in  aircraft  using  any  form  of  large  picture  displays. 
These  displays  are  currently  more  used  in  surveillance  or 
Command  &  Control  type  aircraft,  although  there  is  liable  to  be 
an  increasing  use  of  big  picture  displays  in  rotary  and/or  fixed 
wing  strike  aircraft  as  well.  The  problem  lies  in  the  use  of  a 
cursor  in  a  large,  and  often  cluttered,  display,  where  the  position 
of  the  cursor  on  the  screen  is  not  always  immediately  clear.  For 
small  FOV  displays  (say  20  deg  x  20  deg)  the  cursor  position  can 
be  determined  more  easily  as  it  lies  generally  within  the  foveal 
cone  of  the  eye  and  conventional  manually  controlled  mice  or 
joy-sticks  are  adequate.  In  a  larger  display,  however,  it  can  need 
considerably  more  scanning  to  find  the  cursor  prior  to 
repositioning  it  -  with  the  obvious  time  delays.  With 
conventional  cursor  control,  it  is  necessary  to  find  the  existing 
position  of  the  cursor  in  order  to  know  which  way  to  move  the 
manual  control  to  reposition  the  cursor  at  its  new  point.  By  the 
use  of  eye  tracking,  however,  it  will  be  possible  to  reposition  the 
cursor  by  the  combination  of  fixing  the  eye  on  the  required  point 
and  commanding  the  reposition  with  either  a  manual  control  or 
by  the  use  of  a  voice  command.  This  could  also  be  used  to 
reposition  target  boxes  or  similar  designators  in  large  screen 
displays,  and  combinations  of  eye  tracking  for  coarse  control  and 
manual  for  fine  control  are  feasible  options.  This  combination  of 
eye  designation,  manual  fine  control  and  target  box  labelling  by 
voice  command  -  a  multi-modal  dialogue  -  has  the  potential  to 
provide  not  only  greater  flexibility  in  the  control  of  complex 
display  systems,  but,  potentially,  significant  reductions  in  aircrew 
workload. 

One  useful  spin-off  application  of  viewing  the  eye  in  an  eye 
tracking  system  is  the  potential  capability  of  monitoring  the  eye 
state  (pupil  size,  blink  rate....  etc)  as  part  of  an  Aircraft  Adaptive 
Interface  that  may  be  feasible  in  providing  adaptive  controls, 
displays  and  aircraft  systems  that  respond,  in  some  way,  to  the 
changing  levels  of  aircrew  state. 


1.5  VOICE  CONTROL 

Although  not  quite  as  mature  as  Head  Tracking,  there  have  been 
significant  increases  in  capability  in  the  last  few  years  and  this 
Direct  Voice  Input  (DVI)  technology  now  has  the  potential  to 
enter  service  with  operational  aircraft  in  short  timescales.  The 
technology  is,  at  least  in  speaker  dependent  systems,  at  a  level 
that  will  allow  it  to  compete  successfully  against  manual 
switching  for  a  number  of  operational  tasks.  A  number  of 
experimental  flight  trials  have  been  carried  out,  in  a  number  of 
countries,  and  the  results,  in  terms  of  technical  capability,  are 
essentially  identical  across  countries  in  the  normal  cockpit 
environment.  However,  the  extremes  of  the  operational  envelope 
ie  under  high  ‘g’,  reduce  the  recognition  rates  of  DVI  systems. 
The  ability  to  fly  ‘eyes-out’  whilst  switching  systems  that 
normally  require  a  visual  operation  within  the  cockpit,  can 
provide  a  significant  advantage  in  the  heat  of  operations  and 
enable  reductions  in  crew  workload.  Experimental  flying  has 
shown  DVI  clearly  as  an  enabling  technology,  particularly  in  the 
areas  of  complex  or  time  consuming  switching.  As  an  example, 
radio  channel  selection  or  in  the  areas  of  switching  the  modes  of 
map  displays  or  other  displays,  DVI  has  an  important  role  to  play 
and  it  is  often  the  additional  time-consuming  detail  of  systems 
operation,  which  causes  interference  with  sequential  operational 
tasks,  that  causes  high  workload  for  aircrew.  The  use  of  DVI 
where  the  feedback  of  the  successful  recognition  is  clearly  visible 
or  audible  is  clearly  more  appropriate  for  aircraft  use  at  this  stage 
of  potential  operational  use.  However,  even  where  the  feedback 
loop  is  not  directly  perceived,  DVI  can  be  used  successfully  and 
trials  in  ground  simulators  during  helicopter  tactical  operations 
have  shown  that  digit  strings,  for  instance  waypoint  entries  into 
navigation  computers,  can  be  easily  accomplished  with  audible 
feedback  of  the  recognised  digits  being  fed  to  the  ear  in  real  time. 
This  is  particularly  so  in  the  cases  of  aircrew  who  regularly  listen 
and  communicate  across  a  number  of  radio  nets.  Thus  feedback 
of  the  digit,  word  or  phrase  that  the  system  has  recognised  can  be 
either  overt  in  terms  of  audio  or  visual  feedback  or  covert  in 
terms  of  the  pilot  recognising  that  the  voice  command  has 
actually  changed  the  display  or  system  (e.g.  a  map  display 
changing  scales  from  50,000  to  250,000) 

One  major  advantage  over  manual  hard  or  soft  key  control  is  in 
being  able  to  enter  a,  sometimes  complex,  hierachical  control 
structure  at  any  point.  In  most  current  systems  (navigation, 
attack,  TV-TABS,  etc.)  it  is  necessary  to  page  through  the  levels 
of  a  hierarchical  menu  to  reach  the  level  required.  In  the  RAE 
(now  DERA)  Tornado  flight  trials,  DVI  was  used  on  the 
navigators  TV-TABS  and  it  was  possible  to  access  different 
levels  of  the  navigation  hierarchy  directly  with  potential  time 
savings.  Whilst  later  systems  have  a  less  time  consuming 
approach  to  the  ability  to  access  deeper  parts  of  the  system 
hierarchy,  by  buttons  or  switching,  there  remain  structural 
problems  with  this  approach,  and  whilst  considerable  ingenuity 
has  been  expended  on  reducing  the  number  of  button  presses  to 
access  the  required  information,  only  manual  keyboarding  or 
voice  control  will  allow  direct  access  to  the  functions 

The  acceptance  by  aircrew  of  such  voice  command  systems  has 
been  generally  high  in  all  countries  where  experimental  flying 
has  taken  place.  In  1993  RAE  trials  in  the  UK  in  a  Tornado 
aircraft,  65  %  of  aircrew  thought  that  the  system  was  acceptable, 
with  small  improvements  needed,  25%  rated  the  system  as 
neutral  (ie  neither  good  nor  bad)  and  the  remaining  10%  thought 
it  unacceptable  and  needing  major  improvements.  None, 
however,  thought  it  totally  unacceptable.  Since  those  trials  there 
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Comparison  between  helmet  pointing  and  eye  pointing  envelopes 
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Figure  1-5  Head  and  Eye  tracking  Envelopes 


have  been  improvements  in  the  technology  of  recognition 
systems  and  acceptability  ratings  should  have  improved. 

Similarly  in  the  EF  2000  programme  the  use  of  DVI  in  the  active 
cockpit  simulator  has  shown  the  operational  effectiveness  of 
using  such  systems,  and  the  systems  are  fully  supported  by  pilot 
opinion 

The  extremes  of  the  cockpit  environment,  that  is  extremes  that 
affect  either  the  speech  signal  to  noise  ratios  (eg  noise)  or  the 
effect  of  the  environment  on  speech  production  (eg  high  noise, 
pressure  breathing,  high  ‘g’  etc.)  still  cause  some  problems  with 
speech  recognisers  and  this  is  obviously  more  evident  in  the 
high-performance  fixed  wing  aircraft  than  rotary  wing  aircraft, 
which,  at  this  stage,  makes  DVI  systems  more  suitable  for 
immediate  operational  use  in  helicopters  and  transport  aircraft. 
For  fixed  wing  applications  experimental  research  has  shown  that 
under  increasing  levels  of  g^  relative  speech  levels  increase  by 
up  to  13  to  14  dB  at  8g^ ,  with  a  large  spread,  and  this,  combined 
with  speech  spectrum  changes,  results  in  a  declining  recognition 
performance  from  around  94  to  96%  accuracy  at  the  standard  1  g^ 
to  around  90%  at  Sg,.  The  necessity  to  use  pressure  breathing  at 
the  higher  ‘g’  levels,  also  results  in  reductions  in  recognition 
accuracy,  with  both  ‘g’  and  pressure  breathing,  are  down  to 
around  65%  at  6g,  and  78%  at  5g.  Thus  comparing  pressure¬ 
breathing  and  non  pressure-breathing  conditions  at  5g,  a  loss  of 
some  12%  can  be  attributed  to  pressure-breathing  effects  alone. 

However,  in  spite  of  these  performance  losses  at  the  performance 
extremes,  DVI  systems  work  well  in  the  high  percentage  of 
mission  time  that  the  operational  aircraft  spends  below  5  to  6  g  - 


and  in  a  well  loaded  ground-attack  aircraft  of  the  Tornado  or 
Harrier  type  -  this  is  essentially  the  majority  of  the  time. 

The  further  development  of  speaker  independent  systems  will 
reduce  the  need  for  training  speech  recognition  systems  and 
further  techniques  to  improve  recognition  rates  when  operated 
under  high  environmental  and  battle  stress,  will,  in  the  near 
future,  make  voice  recognition  a  highly  flexible  alternative 
control  technology. 

Voice  control  also  has  potential  for  the  supplementary  aspect  of 
Alternative  Control  Techniques.  In  the  HOTAS  case,  for 
instance,  the  problems  may  lie  in  the  inability  to  remember  either 
the  position  of  the  switch  or  the  name  of  the  function  to  be 
operated  -  more  probably  the  former  than  the  latter.  With  the  use 
of  voice  command  to  switch  the  system,  the  problem  of 
memorising  the  switch  or  button  positions  is  effectively  nullified, 
and  only  the  lesser  problem  of  remembering  the  functions  is  left  - 
in  practice  this  should  significantly  reduce  errors.  Again,  in 
practice,  as  with  most  alternative  control  technologies,  it  would 
be  wise  to  retain  redundancy  in  the  system  and  allow  operation 
by  either  manual  and/or  voice  operated  controls  -  pilot  preference 
being  allowed  depending  upon  sortie  patterns  and  phases.  By 
using  both  systems,  the  number  of  manual  operations  on  the 
HOTAS  controls  could  be  significantly  reduced  and  HOTAS 
used  for  the  time  critical  functions  only,  rather  than  its  current 
potential  for  over-use  -  as  there  are  no  alternative  control 
techniques  to  replace  manual  switching. 

1.6  Biopotential  and  Gesture  based 

Gesture  based  and  biopotential  based  control  are  probably  more 
longer  term,  certainly  in  the  context  of  the  conventional  cockpit. 
There  are  essentially  two  types  of  biopotentials  that  can  be  used 
in  control.  One  is  the  electromyographic  (EMG)  signal  which  is 
associated  with  the  contraction  of  the  skeletal  muscle  and  the 
other  is  the  electroencephalographic  (EEG)  signal  associated  with 
the  brain  activity,  because  these  signals  can  be  modified 
voluntarily  to  indicate  the  intention  of  the  operator.  As  a  control 
modality,  the  principal  objective  is  to  measure  the  biopotential 
activity  from  the  operator  so  that  it  can  be  designated  desired 
control  options  or  augment  other  control  modalities  to  reduce 
selection  ambiguities.  As  an  example  of  the  augmentation  of 
other  control  modalities,  recent  work  [1-10]  has  shown  that  the 
phonetically-relevant  orofacial  motions  can  be  estimated  from  the 
underlying  EMG  activity.  It  is  reasonable  to  assume  that 
information  from  the  EMG  of  facial  muscles  could  improve  the 
performance  of  current  speech  recognition  systems,  particularly 
under  the  adverse  environmental  conditions  found  in  cockpits 
during  operational  flight. 

In  the  area  of  state  monitoring  of  the  operator,  processed  EMG 
and  EEG  signals  can  also  be  used  to  assess  operator  alertness, 
muscle  fatigue  and  workload. 

However,  as  the  cockpit  evolves  from  the  conventional  to  the 
virtual  cockpit,  a  wider  range  of  opportunites  arise.  The  trend 
towards  the  use  of  UAVs  gives  rise  to  the  potential  of  both 
airborne  based  and  ground  based  ‘cockpits’  where  full  or  partial 
control  of  the  UAVs  are  based  on  ‘man-in-the-loop’  principles. 
As  ‘cockpits’  move  away  from  the  harsh  environment  of  existing 
military  aircraft  operations,  to  the  potentially  more  benign 
conditions  of  the  ground  based,  or  the  control-aircraft 
(Civil/Transport  type)  based  cockpit,  the  conditions  for  use  of 
most  alternative  control  technologies  become  more  attractive, 
and  this  is  particularly  so  for  gesture  and  biopotential  controls. 
Environments  in  aircraft  of  the  C^I  type,  AW  ACS.  Rivet  Joint 
etc,  have  control  areas  that  will  be  similar  to  some  airborne  based 


13-9 


cockpits  and  perhaps  are  potentially  the  first  type  of  aircraft  that 
will  be  able  to  make  use  of  these  advanced  alternative  controls. 

1.5  UNMANNED  AIR  VEHICLES  (UAVS) 

Over  the  next  decade  there  is  likely  to  be  an  increasing  transition 
from  air  based  cockpits  to  ground  based  cockpits  for  use  with 
man-in-the-loop  Unmanned  Air  Vehicles  (UAVs).  In  the 
manned  aircraft,  the  trend  is  likely  to  be,  at  least  in  a  large 
number  of  air-to-ground  operations,  to  isolate  the  human  crew,  as 
much  as  possible  from  the  risks  associated  with  combat  areas. 
The  natural  trend,  which  is  already  visible  from  recent  conflicts, 
is  to  produce  stand-off  weapons,  either  autonomous  or  with  a 
man-in-the-loop  control  capability.  Currently  this  is  done  from 
an  airborne  platform  situated  far  enough  from  the  target  to 
minimise  the  risk  of  loss  of,  or  damage  to,  the  aircraft.  As  data 
links  improve,  by  increased  distance,  immunity  to  jamming  and 
increased  bandwidths,  the  controlling  site  will  be  able  to  move  to 
larger  aircraft  platforms  and  finally  to  ground  borne  stations.  In 
each  of  these  ground  stations  (ground  or  air  based),  control  can 
be  of  either  UAVs  which  are  intended  to  fly  returnable  missions  - 
or  UAVs  which  are  not  intended  to  return  to  base. 

Movement  of  the  control  station  to  the  technically,  and 
environmentally,  more  friendly  ground  station  has  a  number  of 

obvious  advantages.  Noise,  vibration,  heat .  and  those 

discomforts  and  partial  disablers  associated  with  aircraft 
manoeuvres  -  high  ‘g’  for  example  -  are  not  present  and  the 
encumbrancies  necessary  for  aircrew  protection  -laser  protection, 
flying  helmet,  oxygen  mask,  ‘g’  suit,  NBC  personal  equipment 
etc.,  -  are  eliminated.  Other  factors,  such  as  displays  equipment, 
do  not  require  the  airborne  equipments  limitations  on  mass  & 
volume  to  be  implemented,  nor  do  associated  issues  such  as 
display  brightness  and  display  power.  This  should  allow 
Commercial  Off  the  Shelf  (COTS)  avionics  equipment  to  be 
more  utilised  which  will  significantly  support  the  affordability  of 
these  type  of  military  operations. 

Consequently,  the  use  of  Alternative  Control  Technologies  to 
supplement  the  natural  human  performance,  often  in  terms  of 
speed  and  accuracy,  rather  than  compensate  for  the  inadequacies 
and  compromises  that  are  essential  in  the  cockpit  environment, 
are  more  viable. 

For  instance,  head-tracking  systems  are  not  exposed  to  unwanted 
motion  from  ground  induced  turbulence  during  ground  attack 
sorties,  voice  system  recognition  rates  improve  in  a  low  noise 
and  vibration  free  environment,  eye  tracking  devices  will  not 
require  the  complex  integration  into  the  airborne  flying  helmet 
and  devices  that  are  sensitive  to  environmental  infra-red 
emissions  (eg  sunlight)  can  be  more  readily  used  -  if  appropriate. 

The  benefits  of  using  alternative  control  technologies  are  not 
only  apparent  in  the  severe  military  air  environment.  The  ability 
to  operate  more  naturally  with  avionic  and  military  systems,  even 
in  the  more  benign  environments  of  the  surveillance  aircraft  or 
the  ground-borne  UMA/UAV  cockpit,  should  provide  significant 
benefits  to  military  operations. 

1.6  CONCLUSIONS 

Future  manned  cockpits  will  inevitably  have  more  complex 
avionic  fits  to  cope  with  more  demanding  operational  scenarios 
and  aircraft  roles,  and  there  will  need  to  be  an  advance  in  the  way 


that  aircrew  interface  with  the  aircraft  systems  in  order  to  enable 
efficient  control  between  man  and  the  rising  complexity  of 
aircraft  systems.  The  number  of  manual  control  systems, 
including  buttons,  keyboards,  and  switches,  is  reaching  a  point 
where  training  aircrew  to  remember  the  phases  and  modes  of 
switching  could  become  both  a  significant  proportion  of 
operational  training  cost  and  also  have  flight  safety  implications. 
Similarly  the  increasing  number  of  switches  on  HOTAS  controls 
has  the  potential  to  heighten  confusion  rather  than  provide 
solutions.  What  is  required  are  alternative  methods  of  inputting 
data  to  aircraft  avionic  systems,  particularly  if  they  provide  a 
more  natural,  and  quicker,  interface.  A  simple  example  of  this  is 
in  the  use  of  voice  input  as  an  alternative  to  remembering  and 
dialling  up  radio  frequencies.  A  single  command  phrase  - 
Famborough  Tower  -  for  instance,  replaces,  essentially,  a  three 
segment  approach  -  remember  frequency,  dial  frequency  and  call 
controller  on  that  frequency.  Of  the  more  mature  alternative 
control  technologies,  voice  recognition  and  head  tracking  are 
both  in  operational  flight  and  experimental  flight  -  depending 
upon  the  level  of  sophistication  of  the  technology  -  and  are  both 
technically  mature  enough  for  full  operational  use,  with  research 
on  the  next  generation,  higher  capability,  systems  in  progress. 

Eye  based  control  is  laboratory  mature,  and  used  for  assessing 
eye  movement  in  simulators,  and,  with  development,  has  the 
potential  to  integrate  effectively  in  the  operational  environment 
with  head  and  voice  based  control.  Gesture  and  biopotential  are 
probably  the  least  mature,  but  provide  potential  for  the  longer 
term  aircraft  systems  (2020)  and  may  be  particularly  of  use  in 
ground  based  cockpits  of  man-in-the-loop  UAVs. 

All  systems  in  a  civil  and  military  aircraft  must  provide  some 
tangible  operational  benefit  -  particularly  in  retrofit  cases  -  and 
both  head  and  voice  based  control  are  expected  to  provide  that 
benefit  in  the  third  generation  aircraft  (Eurofighter  and  Rafale). 
This  would  be  supplemented,  in  due  course,  with  eye  based 
control,  particularly  in  the  air-to-air  engagement  role,  but,  also,  to 
a  lesser  extent,  in  the  air-to-ground  role. 

The  benefits  of  alternative  control  techniques  lie  in  a  more 
natural  interface  with  the  aircraft,  improved  speed  of  operation 
and  reduction  in  training  overheads. 

Released  from  the  constraint  of  only  one  communication  channel 
with  the  aircraft  systems  -  manual  -  the  use  of  alternative  control 
technology  invites  aircrew,  aircraft  and  systems  designers,  and 
others,  to  be  more  imaginative  in  their  interaction  with  the 
aircraft  and  systems,  using  these  alternative  controls  as 
appropriate  to  the  operational  benefits  and  needs.  Such 
alternatives  are  not  intended  primarily  to  replace  manual  controls 
but  to  supplement  manual  systems  and  to  provide  alternatives,  to 
be  used  as  the  occasion  requires.  Aircraft  systems,  however, 
need  to  be  practical,  to  retain  as  simple  an  interface  as  the 
technological  complexity  of  the  systems  allows  and  be  operated 
by  aircrew  with  a  wide  range  of  capabilities.  This  should  ensure 
that  the  use  of  these  alternative  controls  is  balanced  by  the 
aircraft  designers  natural,  and  often  historically  justified,  inherent 
scepticism  of  the  useability  of  new  technologies. 
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1.  SUMMARY 

Microelectronics  have  forced  their  way  into  military  air¬ 
craft-  and  airliner-cockpits  more  than  a  decade  ago.  On  the 
military  side  it  was  physical  contest,  on  the  civil  side,  eco¬ 
nomic  competition  was  the  drive  leading  the  evolution  from 
the  analogue  to  the  digital  cockpit. 

Regarding  today’s  state-of-the-art  military  and  civil  trans¬ 
portation  aircraft,  one  can  hardly  find  any  information  on 
pilots’  wish-list,  that  can  not  be  picked  up  by  a  sensor  al¬ 
ready  installed.  Despite  of  this  technical  ability  to  seize  the 
aircraft  in  its  full  complexity,  there  is  obviously  still  some¬ 
thing  wrong  according  to  the  aviation  accident  statistics. 
‘Human-factors’  often  is  the  ultimately  found  result  in  the 
search  for  crash  reasons.  But  this  conclusion’s  correctness 
depends  on  the  definition  of  ‘human-factors’. 

It  is  a  well-known  fact,  that  even  highly  trained  professional 
pilots’  abilities  can  easily  be  reduced  to  amateur  status 
under  excessive  mental  stress.  In  these  situations  the  human 
mind  only  accepts  intuitively  perceived  information  as  basis 
for  behaviour.  Therefore  it  is  vital,  that  new  avionics  with 
their  inert  tendency  to  become  complex,  are  carefully  de¬ 
veloped  along  the  principles  of  intuitive  use. 

This  paper  describes  several  parts  of  a  project  comprising 
the  development  of  a  new  cockpit  for  General  Aviation 
aircraft.  It  focuses  on  aspects  of  the  target  group,  design- 
philosophy  and  low-cost  realisation. 

2.  INTRODUCTION 

Both  military  and  civil  commercial  aviation  development  is 
driven  by  competition.  Where  it  is  a  rather  physical  one  on 
the  military  side,  it  is  economic  contest  on  the  civil  side: 

The  world  wide  inclining  need  for  transportation  results 
into  a  continuous  increase  of  the  aviation  branch.  Especially 
the  last  two  decades  saw  civil  aviation  traffic  grow  by  the 
faetor  of  five  [1]. 

With  the  positive  development  of  this  economical  branch, 
an  increasing  number  of  airlines  were  foimded,  competing 
on  safe  and  fast  arrival  on  defined  destinations.  But  the 
growing  number  of  participants  in  air  traffic  and  the  in¬ 
creasing  air-speeds  raised  the  risk  of  mid-air  collisions.  To 
keep  up  flight-safety  an  air  traffic  control  system  was 
erected.  For  the  operation  of  the  system,  precise  locating 
and  navigation  was  necessary,  as  well  as  reliable  communi¬ 
cation  between  air-  and  ground-stations. 

Forced  by  the  competition  the  airlines  wished  to  run  their 
aircraft  in  the  most  economic  way.  Beside  safety-relevant 
information,  an  ever  increasing  number  of  additional  sys¬ 
tem-data  became  important  for  aircraft-operation. 


From  this  point  of  view,  the  man-machine-interface  ‘cock¬ 
pit’  can  be  divided  into  five  functional  groups:  in-flight 
conditions/configuration,  engine,  subsystems,  locating/ 
navigation  and  communication. 

Regarding  communication,  locating/navigation,  engine  and 
subsystems,  one  can  describe  several  technological  devel¬ 
opments  used  to  increase  safety  and  economical  quality  of 
aircraft.  Most  of  them  were  invented  for  military  applica¬ 
tions,  but  were  quickly  transferred  into  Commercial  Avia¬ 
tion  due  to  their  usefulness. 

In  the  field  of  communication,  several  procedures  operating 
on  radio-waves  were  developed  after  1930.  In  the  beginning 
one  used  Morse-code,  later  language,  nowadays  Informa¬ 
tion  can  be  exchanged  digitally  (e.g.:  ADS-B). 

In  the  beginning  of  navigation,  landmarks  were  used  for 
orientation,  followed  by  astro-navigation  and  dead  reckon¬ 
ing.  With  the  increasing  quality  of  micro-mechanics,  gyros 
were  used  for  inertial  navigation  and  finally  radio  beacons 
(e.g.:  VOR,  DME)  were  set  up.  With  two  stations  in  range, 
the  position  of  the  aircraft  could  be  plotted,  using  an  on¬ 
board  receiver.  At  last  satellite  navigation  systems  were 
deployed  in  the  80’s.  The  US-American  GPS  provides  users 
with  three-dimensional  position  data.  The  almost  world 
wide  service  is  independent  from  time  and  weather- 
conditions  and  is  precise  within  100  m  for  civil  users.  Using 
several  correction  techniques,  precision  within  sub-meter- 
range  can  be  realised  [2]. 

Regarding  engine  and  subsystem  surveillance  the  cockpit 
instrumentation  was  confined  on  ‘engine  revolutions  per 
minute’  and  ‘fuel  quantity’  up  to  WW  II.  Only  the  use  of 
ever  more  sophisticated  piston  engines  and  turbines  on  the 
one  hand  and  the  deployment  of  electric  and  hydraulic 
subsystems  on  the  other  hand  required  the  increased  re¬ 
cording  of  relevant  data. 

The  usual  integration  procedure  of  new  cockpit  devices  was 
an  addition.  Substitution  of  systems  was  carried  out 
sparsely  and  thus,  the  complete  system  ‘aircraft’  became 
more  and  more  complex.  The  cockpit  evolution  resulted 
into  the  extension  of  training  and  the  increase  of  number  of 
the  cockpit-staff 

Starting  with  one  pilot  and  peaking  five  members  in  the 
early  sixties,  the  cockpit  crew  was  reduced  to  pilot,  first 
officer  and  flight-engineer  until  the  late  seventies  due  to  the 
automation  of  several  subsystems  (figure  1).  Extensive 
developments  in  microelectronics  allowed  a  further  reduc¬ 
tion  down  to  two  cockpit  crew  members  in  the  mid-eighties 
(figure  2). 
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Figure  1:  Sud-Aviation  Caravelle  cockpit  (1957). 
Source:  Aerokurier 


Figure  2:  Airbus  A340  cockpit  (1997).  Source:  Airbus  Ind. 

The  use  of  computers  and  electronic  displays  opened  an¬ 
other  complete  step  in  automation.  Sophisticated  electronic 
systems  (e.g.  EFIS,  EICAS,  FMS)  carry  out  complex  pro¬ 
cedures  of  planing  and  surveillance.  The  man-machine 
communication,  necessary  therefore,  is  performed  by  using 
the  multi-functional  input-  and  display-devices. 

The  connection  between  technological  progress  and  cockpit 
design  is  obvious.  Especially  developments  in  microelec¬ 
tronics  caused  the  last  distinct  change-in-trends. 

3.  GEPiERAL  AVIATION  COCKPIT  DESIGN 

The  term  ‘General  Aviation’  entitles  a  special  branch  of 
aviation.  Its  existence  is  obvious,  but  its  description  is 
difficult,  because  neither  the  pilots,  nor  the  aircraft  in¬ 
cluded,  share  a  common  characteristic. 

As  a  consequence,  there  is  no  specific  definition  for  this 
term.  According  to  [3]  it  may  usefully  be  described  through 
an  exclusion:  General  Aviation  (GA)  includes  those  areas  in 
aviation,  being  not  military-  or  airline-Zcharter-aviation. 

The  term  ‘General  Aviation’  was  created  in  the  sixties, 
when  there  was  a  significant  increase  in  numbers  of  aircraft. 
Nowadays,  GA  has  a  great  significance  compared  to  other 
branches  in  aviation.  About  90%  of  world  wide  registered 
civil  aircraft  are  reckoned  among  GA.  With  those  aircraft 
75%  of  in-flight  hours  are  produced,  50%  of  the  passengers 
are  transported,  but  only  7%  of  fuel  is  consumed  world 


wide.  Most  of  the  flights  in  GA  are  carried  out  under  visual 
flight  rules  (VFR).  In  this  specific  sort  of  flight,  the  attitude 
of  the  aircraft  and  traffic  co-ordination  is  controlled  through 
vision. 

More  than  80%  of  GA  aircraft  are  powered  by  a  single 
piston  engine,  include  up  to  four  seats,  and  have  a  maxi¬ 
mum  take-off  weight  up  to  5.7  t.  More  than  60%  of  the 
world  wide  GA  aircraft  are  registered  in  the  USA,  the  aver¬ 
age  age  of  all  GA  aircraft  is  about  20  years  [1]. 

The  world  wide  number  of  GA  pilots  is  estimated  about  1.4 
Million  [1].  The  statistics  regarding  age,  total  in-flight 
hours  and  in-flight  hours  per  year  show  a  high  standard 
deviation.  Therefore  the  forming  of  a  mean  value,  allowing 
a  statement  relating  to  physical  capability,  experience  and 
training  condition  is  pointless.  This  fact  rather  outlines  the 
picture  of  the  extreme  heterogeneity  among  GA  pilots. 
Designing  for  this  target  group  means  to  take  into  consid¬ 
eration  both,  the  professional’s  claim  as  well  as  the  ama¬ 
teur’s  abilities. 

Comparing  Commercial  and  General  Aviation  cockpits, 
only  the  first  periods  of  the  evolution,  described  above  is 
detectable  in  the  field  of  GA.  As  shown  in  figures  3  and  4, 
there  are  no  fundamental  differences  obvious  between  the 
instruments  featured  in  both  eockpits.  The  number  of  input- 
and  display-devices  has  doubled  in  the  last  40  years.  Auto¬ 
mation  and  deployment  of  computers  or  electronic  displays 
does  virtually  not  occur. 


Figure  3:  Beech  Bonanza  cockpit  (1957).  Source:  Aerokurier 


Figure  4:  Beech  Bonanza  cockpit  (1997).  Source:  Aerokurier 
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Nevertheless  do  GA  pilots  likewise  wish  to  take  the  advan¬ 
tages  of  the  new  technologies.  As  a  result,  aircraft  were 
equipped  with  degraded  Commercial  Aviation  (CA)  pro¬ 
ducts  (e.g.  NAV-radio,  GPS-receiver).  These  devices  have 
their  own  specific  operation-strategies  and  thus,  are  often 
not  suitable  for  the  use  in  GA.  Furthermore,  the  imple¬ 
mented  variety  of  functions  as  well  as  the  means  of  com¬ 
munication  between  device  and  user  is  often  inadequate  for 
non-professional  pilots.  Especially  under  high-workload 
conditions  or  in  critical  phases  of  flight  the  pilot  may  be 
overstrained. 

In  addition,  one  can  detect  that  the  great  number  of  input- 
and  display-devices  results  into  installation  problems.  Only 
a  few  displays  are  located  within  the  pilot’s  central  field  of 
vision  and  several  input-devices  can  only  be  reached  under 
extensive  movements  of  the  pilot. 

The  great  number  of  displays,  their  eccentric  position  and 
therefore  poor  readability  force  the  pilot  into  long  ‘head- 
down  flying’  periods.  Especially  on  VFR  flights  this  state 
must  be  regarded  as  extremely  safety-critical. 

The  reason  for  the  non-appearance  of  the  last  step  of  evolu¬ 
tion  in  GA  cockpit  design  is  the  cost-factor.  An  empirical 
coefficient  is,  that  up  to  ten  percent  of  a  GA-aircraft’s 
worth  may  be  spend  for  avionics.  The  facts  are,  that  one  has 
to  spend  as  much  for  a  CRT  (cathode-ray  tube)  used  for  a 
display  in  the  Airbus  A340  and  the  necessary  periphery  as 
about  for  a  used  Cessna  172  (a  typical  GA-aircraft).  In  view 
of  these  two  figures  it  becomes  clear,  that  beside  the  fact 
that  those  CRTs  are  too  large  and  heavy  to  be  installed,  it 
would  economically  make  no  sense  to  base  the  development 
of  a  new  cockpit  on  this  technology. 

As  the  total  costs  result  out  of  development,  certification 
and  production  costs,  especially  the  down-headed  trend  in 
production  costs  of  micro-electronic  devices  suggest  the 
introduction  of  new  technologies  into  GA. 


4.  The  COSIMA  -  PROGRAM 

In  view  of  this  situation,  a  team-program  featuring  several 
mid-sized  enterprises  and  the  Institut  fur  Flugfuhrung  at  the 
Technische  Universitat  Braunschweig  was  initiated  in  Janu¬ 
ary  1996.  The  COSIMA-program  (Cockpit  and  Simulator 
for  General  Aviation)  is  sponsored  through  the  DLR  (Ger¬ 
man  Aerospace  Centre)  by  the  German  Ministry  of  Eco¬ 
nomics  as  part  of  the  aviation  research. 


Figure  5:  COSIMA-cockpit  implemented  in  the  C-1 72-simulator 
fuselage. 


The  schedule  shows  three  different  phases  within  the  pro¬ 
gram.  In  the  beginning,  the  basic  cockpit-concept  was  de¬ 
signed.  In  the  second  step,  the  concept  was  realised  in  a 
research-simulator  (based  on  an  original  Cessna  C-1 72 
fuselage)  for  testing  purposes.  For  the  now-running  third 
step,  the  cockpit  was  implemented  in  a  real  C-1 72  and 
undergoes  excessive  in-flight  testing  necessary  for  certifi¬ 
cation  [4]. 


Figure  6:  COSIMA-cockpit  implemented  in  the  C-1 72-testbed. 


4.1  Basic  Structure 

As  described  above,  the  cost-factor  is  of  great  importance 
for  the  program.  As  a  consequence,  the  concept  has  a 
modular  layout  and  can  therefore  be  easily  adapted  to  a 
variety  of  GA  aircraft.  Furthermore,  system  components 
may  be  separated  out  and  can  be  implemented  as  subsys¬ 
tems  into  conventional  cockpits.  Last  but  not  least  in  case 
of  a  failure  or  damage  of  one  module,  it  can  easily  be  re¬ 
placed.  First,  the  concept  was  designed  for  VFR-flying,  an 
IFR  upgrade  is  plaimed  (IFR:  Instrument  Flight  Rules). 

The  main  purpose  of  the  new  cockpit-concept  is  the  inte¬ 
gration  of  new  technologies,  considering  the  specific  prem¬ 
ises  in  GA.  Therefore  the  extent  of  implemented  functions, 
the  operational  logic  of  subsystems,  the  graphic  user- 
surfaces  and  the  three-dimensional  environment  of  the  pilot 
have  to  be  taken  into  account  for  the  new  design. 

The  basis  for  the  design  process  is  founded  through  opera¬ 
tion  analysis  carried  out  in  several  GA  cockpits. 

The  system  features  two  PC-like  computers,  operating  three 
LC-Colour-Displays  (figure  7).  The  input  is  performed 
through  a  five  button  keyboard  and  a  track-ball-like  graphic 
input-device. 


Figure  7:  Schematic  structure  of  the  electrical  system  compo¬ 
nents. 
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The  main  task  of  the  first  computer  is  to  record  data  of 
several  different  sensors  (engine-sensors,  GPS-sensor,  etc.) 
and  to  generate  the  output  on  the  centre-display  (CD).  The 
first  computer’s  performance  may  be  compared  to  a  100- 
MHz  Pentium  PC.  The  second  computer  processes  the 
provided  data  and  performs  the  output  on  the  displays  on 
the  left  and  right  side  (SD).  The  second  computer’s  per¬ 
formance  must  be  higher  for  best  graphic  results. 

The  computers  are  supplied  by  redundant  sources  of  elec¬ 
tricity  and  perform  a  permanent  surveillance  of  both  their 
operational  status.  In  case  of  a  serious  disturbance,  the 
system  switches  to  ‘emergency-mode’.  In  this  secured 
mode  the  pilot  is  provided  only  with  relevant  basic  instru¬ 
mentation. 


4.2  Basic  Physical  Layout 

A  basic  design  principle  is  the  definition  of  areas  or  space 
for  input-  and  display  devices.  The  pilot  should  be  able  to 
identify  areas  of  different  context  easily.  The  aircraft  should 
be  controlled  out  of  the  left  or  right  seat,  as  it  might  be  used 
for  training  purposes.  This  premise  leads  to  the  concentra¬ 
tion  of  single-appearing  input-  and  display-devices  on  the 
longitudinal  centre  axis  of  the  cockpit  between  the  front 
seats. 

Each  side-display  is  placed  in  front  of  the  seats,  whether  the 
third  display  is  fixed  in  a  middle-position  (figure  8). 


Figure  8:  COSIMA  cockpit  concept:  physical  layout. 

The  cockpit  consists  of  three  main  segments.  The  middle- 
segment  practically  contains  all  major  input-devices  (except 
for  steering  wheel/  control  column  and  pedals).  Its  shape 
results  out  of  the  longitudinal  ergo-cinematic  curve  that  is 
virtually  drawn  by  the  fingertips  of  a  human  being,  trying  to 
reach  all  the  input  devices.  The  surface  orientation  is  de¬ 
signed  to  give  the  pilot  perfect  view  angles  on  any  device 
installed  (three-dimensional  devices  unequal  90°,  flat  input- 
or  display-devices  about  90°). 

In  the  remotest,  almost  upright  part  of  the  middle-segment, 
the  centre-display  for  joint  use  is  fixed.  In  this  place  it  has  a 
perfect  position  for  the  fast  switching  of  view  between 
display  and  outside  airspace.  Furthermore,  the  Increased 
distance  from  eye  to  display  makes  the  eye’s  near-far- 
accomodation  easier  for  the  pilot.  On  this  screen,  the  ‘Air¬ 
craft  Monitoring  System’  is  displayed. 


The  side-segments  mainly  serve  as  housings  for  the  side- 
displays  (and  possibly  for  the  steering  linkage).  The  side- 
displays  are  fixed  in  the  upper  part  of  the  segment  and 
remotely  from  the  pilot’s  eye,  corresponding  to  the  centre- 
display  mounting.  On  these  screens,  the  in-flight  conditions 
and  the  Flight  Planing  and  Navigation  System  are  dis¬ 
played. 

By  giving  the  cockpit  this  specific  shape,  all  hard-surface 
and  glass  objects  are  away  from  typical  crash-related 
movements  of  the  pilot’s  head  and  upper  body,  and  thus  the 
injury-risk  is  significantly  decreased  (figure  9). 


Figure  9:  Sketch  of  the  new  cockpit  concept,  showing  ergo¬ 
nomic  details  (side  view). 

4.3  The  Aircraft  Monitoring  System  (AMS) 

The  AMS  contains  the  engine  and  electrical  subsystem  data. 
The  following  information  is  displayed:  engine  revolutions 
per  minute  (r.p.m.),  oil  pressure,  oil  temperature,  exhaust 
gas  temperature  (EGT),  cylinder  head  temperature  (CHT), 
battery-voltage,  generator-amperes  and  time. 

Oil  pressure  and  oil  temperature,  EGT  and  CHT,  as  well  as 
battery-voltage  and  generator-amperes  are  automatically 
monitored  and  displayed  in  groups  (figure  10). 


Figure  10:  AMS-display  in  main  level  status  with  keyboard. 
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By  using  the  keyboard,  any  individual  information  can  be 
displayed  explicitly  and  the  pilot  is  not  excluded  by  the 
automation  from  the  ‘information-loop’. 

Engine  revolutions  are  still  displayed  as  a  circular-instru¬ 
ment,  dominating  the  AMS-screen.  A  further  important 
component  of  the  AMS  is  the  fuel-management.  The  con¬ 
nection  between  fuel  and  time  is  expressed  through  the 
‘fuel-time-remain’  display. 

The  AMS  has  a  four-level  lay-out.  Each  level  can  be  se¬ 
lected  by  using  the  appropriate  button  on  the  keyboard 
(figure  11). 
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Figure  11:  Typical  operationai  process  in  the  AMS  system- 
hierarchy. 

The  surveillance-status  of  the  system-groups  is  displayed  in 
the  main  level.  In  the  detail  level  the  information  within 
one  specific  group  is  graphically  and  digitally  indicated.  In 
the  proceeding-level  I  and  II  the  temporal  development  of 
one  specific  information  is  indicated.  Proceeding-level  I 
shows  the  time  from  the  point  of  engine  start-up,  proceed¬ 
ing-level  II  focuses  only  on  one  fourth-section  (‘SEC’)  of 
that  time. 

The  pilot  switches  levels  by  using  the  keyboard,  installed 
under  the  screen.  The  shape  and  the  position  of  the  switches 
generate  a  clear  connection  to  the  symbols  presented  on  the 
screen.  The  user  may  return  from  any  sub-level  to  the  main- 
level  by  choosing  the  always  available  ‘go-back’  function. 

In  case  of  a  measured  value  turning  to  ‘Amber’,  the  display 
remains  in  the  main-level.  The  icon,  representing  the  group, 
containing  the  specific  measured  value  turns  amber  likewise 
and  a  detailed  trend-report  is  displayed  along  with  an 
acoustic  warning  signal. 

In  case  of  a  measured  value  turning  to  ‘Red’,  the  display 
automatically  switches  to  the  detail-level  after  an  acoustic 
warning-signal  and  thus  the  specific  measured  value  is 
explicitly  displayed. 


ItO-TIIVIEUTC  09:11 1 
[inflight  02:23 1 


FUEL  TIME  REMAIN 

02:  35+30 


Figure  12:  AMS-display  in  proceeding-level  I  status  (oil  pressure 
turned  'amber'). 

If  more  than  one  value  turns  Amber  or  Red  a  situation- 
adaptive  hierarchy  is  generated  within  the  system  according 
to  the  priority  of  action.  Following  this  hierarchy  only  the 
most  urgent  information  is  displayed  at  a  time. 

Thus  the  AMS  enables  the  pilot  to  focus  on  the  temporarily 
essential  and  relevant  information.  A  complete  system- 
check  may  be  performed  with  a  glance  on  the  centre- 
display.  The  automated  surveillance  decreases  the  pilot’s 
workload,  but  its  permanent-accessible  design  doesn’t  lock 
him  out  of  the  information-loop.  Furthermore,  any  user  is 
able  to  realise  trends  in  measured  values,  just  by  switching 
to  the  proceeding-levels.  This  function  enables  pilots  to  do 
a  ‘trouble-shooting’  on  any  system  of  the  aircraft  and  offers 
means  for  strategic  planing  of  the  further  flight  under  fail¬ 
ure-conditions.  In  the  past  only  exceedingly  experienced 
pilots  were  able  to  perform  this  task  and  only  under  low 
workload  conditions. 

The  design  of  the  AMS-symbols  follows  several  criteria, 
like  intuitive  perception  (fuel  quantity  indicators  on  left  an 
right  side,  separated  by  the  circularly-shaped  r.p.m.- 
indicator),  tradition  (circular  shape  of  r.p.m.-indicator)  and 
regulation  (colour-coding).  This  results  into  a  permanently 
decreased  perceptual  effort  and  almost  zero-hour  training 
periods  for  ratings  on  the  new  cockpit. 


4.4  Flight  Planing  and  Navigation  System  (FPNS) 

All  in-flight  conditions  and  navigational  data  are  presented 
on  the  side-displays.  In  particular,  the  following  informa¬ 
tion  is  indicated:  indicated  air  speed  (IAS),  temporal  change 
of  LAS  (speed-trend),  ground  speed,  barometric  altitude 
(ALT),  temporal  change  of  ALT  (variometer),  compass 
heading  and  ground  track.  Furthermore,  a  map  and  the 
planned  flight-track  is  displayed  (figure  13). 

Whereas  speeds  and  altitudes  are  directly  indicated,  a  spe¬ 
cial  way  of  presentation  was  designed  for  navigation-tasks, 
to  enable  the  pilot  to  handle  the  amount  of  information. 
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Figure  13:  FPNS  on  side-displays  (flight-mode  status). 

Normally,  a  flight  preparation  is  supposed  to  be  created 
with  special  software  on  a  separate  computer  some  time 
before  the  flight.  Using  an  ordinary  disc,  the  flight  log  is 
transferred  to  the  board-computer  of  the  aircraft.  Excep¬ 
tionally  a  flight  preparation  may  as  well  be  generated  in  the 
cockpit  right  before  the  flight. 

The  navigation  system  features  a  true  moving  map.  The 
actual  position  of  the  aircraft,  as  well  as  the  planned  flight- 
track  are  permanently  displayed  on  the  map.  The  flight- 
track  is  defined  through  way-points.  Each  way-point  and 
each  way-section,  enclosed  by  a  pair  of  way-points  may  be 
connected  manually  or  automatically  (using  prepared  data¬ 
bases)  with  specific  additional  data  (e.g.  en-route  frequen¬ 
cies,  airfield  information,  planned  cruising-altitude,  etc.). 

Operations  on  graphic-display  units  inevitably  demand  a 
graphic-input-device  (GID)  for  optimal  intuitive  use.  The 
device’s  function  should  be  way-proportional  and  not 
speed-proportional.  Therefore,  a  ‘track-balT-like  device 
was  specially  designed  for  use  in  aircraft  (figure  14). 


Beside  the  graphic-positioning  device,  it  features  three  main 
functional  units.  The  primary-selection  function  is  realised 
through  a  two-button  solution.  The  buttons’  linear  push-axis 
are  arranged  out  of  main  acceleration-axis  of  the  aircraft 
and  contrary  to  one-another.  This  makes  accidental  manual 
release  or  release  by,  e.g.  a  touch-down-shock  almost  im¬ 
possible. 

Furthermore  there  are  two  zoom-buttons,  to  be  operated 
with  thumbs  (each  one  for  left-  or  right-hand  use)  and  one 
switch,  to  select  Planing  or  Flight  Mode. 

The  GID  is  used  for  all  FPNS  input  operations  (s.b.).  It 
works  with  a  defined  fuzzy-area.  That  means,  the  system 
presumes  the  user  to  have  tried  to  graphically  mark  a  spe¬ 
cific  point  (e.g.  an  airfield  on  the  map)  if  he  actually  hit 
within  the  fuzzy-area  of  that  point. 

In  this  case,  the  FPNS  offers  the  user  a  choice  of  the  actu¬ 
ally  marked  point  and  alternatively  all  specific  points  whose 
fuzzy-areas  have  been  hit. 

This  special  software  design  excessively  increases  the 
operational  precision  and  efficiency  of  the  graphic  input 
device. 

The  FPNS  may  be  operated  in  two  different  modes  (figure 
15). 


Mode-Selection 


Figure  15:  Basic  logical  structure  of  the  FPNS. 

The  Flight-Mode  (FM)  screen  shows  all  in-flight  condi¬ 
tions,  the  map  and  the  flight-log  strip,  containing  way- 
points  and  -sections  and  also  several  display-optional  func¬ 
tions  (s.b).  The  FM  works  on  a  defined,  invariable  flight 
preparation.  The  only  options  within  the  flight  log  are  to 
skip  certain  way-points  or  to  re-activate  them.  Several 
operational  options  are  implemented,  to  adjust  the  display 
to  the  preferences  of  the  user  (s.b.). 

In  the  Planing  Mode  (PM),  flight  preparations  may  be 
created  or  already  existing  logs  may  be  altered.  After  hav¬ 
ing  created  a  new  flight  preparation,  it  may  substitute  the 
operational  one  at  any  time. 

A  ‘zoom’-function,  featured  in  both  modes  enables  the  pilot 
to  get  an  overall  view  of  the  aircraft’s  geographical  envi¬ 
ronment,  as  well  as  a  detailed  impression  of  its  immediate 
surrounding.  Furthermore,  several  different  kinds  of  maps 
may  be  chosen  with  respect  to  the  actual  flight-task,  if 
available  for  that  area  (ICAO-map,  visual  approach  chart, 
etc.).  Several  further  settings  may  be  made  in  accordance  to 
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the  users  preferences  (e.g.:  map  display  in  heading-up  or 
north-up  mode). 

To  avoid  any  graphic  interference  with  in-flight  condition 
indicators,  the  movements  of  the  graphic-input-symbol 
(GIS)  on  the  screen  are  confined  to  the  map-area  and  the 
rectangular  flight-log  area  on  the  bottom  of  the  screen.  A 
potentially  activated  functional  field  on  the  display  is 
graphically  emphasised  to  clarify  the  consequences  of  the 
platmed  action. 


Figure  16:  FPNS  with  active  ‘side-view’  and  circles  of  equal 
flight-time  in  the  map  area  (flight-mode  status). 

Using  the  ‘side-view’  function,  the  third  dimension  of  the 
planned  flight-path  is  displayed  in  the  flight-log  area.  The 
side  view  shows  ground  elevation,  specific  obstacles,  the 
planned  flying  altitude  and  the  aircraft’s  actual  altitude  and 
position  along  the  planned  track.  By  displaying  the  third 
dimension  of  the  actual  air  space  structure,  the  situation 
awareness  of  the  pilot  is  substantially  improved  (figure  16). 
With  air-  and  ground-speeds,  heading  and  ground  track,  the 
board  computer  can  estimate  the  actual  wind-vector  in 
speed  and  direction.  This  information  can  be  used  to  show 
points  of  equal  flight-time  on  the  map  display  (in  general 
non-concentric  circles)  instead  of  points  of  equal  distance 
(concentric  circles). 

If  an  airfield  is  chosen  as  a  way-point,  all  available  special 
maps  are  offered  in  the  corresponding  zoom-level,  near  that 
point.  Due  to  the  unforeseeable  situation  no  planned  flight- 
path  is  displayed  with  those  maps.  In  this  case,  the  side- 
view  function  is  of  great  importance  to  gain  a  three- 
dimensional  overall  view  of  the  situation. 

The  presented  FPNS  decreases  the  workload  for  keeping  the 
aircraft  on  the  path  and  for  navigation  tasks  compared  to 
conventional  cockpits.  Especially  the  three  dimensional 


graphical  display  of  actual  position  and  heading  in  compari¬ 
son  to  the  planned  track  makes  orientation  easier  and  sets 
free  mental  capacity.  The  system  has  a  low  degree  of  com¬ 
plexity  and  is  adapted  to  the  specific  requirements  of  VFR 
flights. 

3.4  Integration  of  input  devices  into  the  cockpit 
The  centre-panel  has  the  affiliated  keyboard  mounted  di¬ 
rectly  under  the  centre-display.  It  can  easily  be  reached  with 
one  arm  stretched  out.  The  keyboard’s  design  allows  the 
fingers  to  fix  the  position  of  the  hand,  in  order  to  facilitate  a 
safe  operational  process.  The  graphic  input-device  for  the 
side-displays  is  installed  on  the  level  part  of  the  centre- 
panel.  For  reliable  operation,  the  armrest  and  bearing  sur¬ 
face  for  the  palm  of  the  hand  may  be  used. 

The  partially  revised  shape  of  input-devices  (e.g.:  semantic 
connection  of  the  flap-lever)  and  their  special  arrangement 
(e.g.:  orientation  of  operational  processes  on  electrical 
switches)  support  processes  of  intuitive  acting  and  motorial 
learning. 

4.  SUMMARY 

The  presented  cockpit-concept  makes  the  opportunities  of 
several  new  technologies  accessible  for  pilots,  flying  GA 
aircraft.  Furthermore  it  increases  the  safety  in  aviation  by 
optimising  processes  of  perception  and  conduct  within 
aircraft  operation.  The  system  layout  is  open  to  any  future 
digital  air-traffic-management-system  and  provides  the 
necessary  tool  to  participate  even  for  non-professional 
pilots. 

The  permanent  information  flow  is  reduced  by  a  balanced 
lay-out  and  an  adjusted  variety  of  functions,  appropriately 
processed  information  and  an  adaptation  of  input-  and 
display-devices  on  the  characteristics  of  GA  pilots.  The 
specific  software  layout  keeps  the  pilot  in  the  information- 
loop  and  creates  a  useful  transparency  of  the  entire  system 
‘aircraft’. 

By  taking  into  account  the  great  individual  differences 
between  GA  pilots’  skills  and  giving  the  cockpit  a  modular 
lay-out,  the  concept  is  suitable  for  a  great  number  of  appli¬ 
cations.  This  is  besides  the  low-cost  availability  the  basis 
for  the  successful  introduction  of  new  technologies  into 
GA. 
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Summary 

The  Flight  Research  Laboratory  (FRL)  of  the 
National  Research  Council  (NRC)  in  co-operation 
with  the  Department  of  National  Defence  and 
industrial  collaborators  Canadian  Marconi  Company 
(CMC)  and  CAE  Electronics  Ltd.  (CAE)  is  working 
to  integrate  new  cockpit  technologies  to  improve 
mission  effectiveness  and  system  safety  of  Search 
and  Rescue  (SAR)  missions  conducted  from 
helicopters. 

Search  and  Rescue  aircraft  operate  in  a  demanding 
environment,  often  in  remote  areas,  at  night,  or  in 
inclement  weather.  Cockpit  systems  that  reduce  pilot 
workload  and  improve  pilot  situational  awareness  can 
save  lives  when  appropriately  integrated  into  the 
aircraft. 

NRC  and  partners  are  building,  integrating  and 
conducting  research  on  an  enhanced  and  synthetic 
vision  system  (ESVS)  to  help  SAR  helicopter  pilots 
complete  their  missions  in  degraded  visual 
environments.  The  ESVS  will  provide  SAR  pilots 
with  a  real-time  display  that  mimics  visual  flight 
rules  (VFR)  conditions. 

NRC  plans  to  integrate  and  evaluate  a  prototype 
system  by  the  year  2000.  The  prototype  system  will 
include  a  visually  coupled  helmet  mounted  display 
(HMD)  system,  a  synthetic  image  generated  from  a 
terrain  database,  an  advanced  sensor  and  an  image 
fusion  system.  A  complementary  research  program 
is  under  way  at  NRC  to  investigate  fundamental 
human-machine  interface  issues  relevant  to  the 
proposed  prototype  system 


Introduction 

Search  and  Rescue  missions  are  difficult  to  carry  out 
at  the  best  of  times,  and  especially  difficult  over 
unknown  terrain  in  the  dark  or  during  inclement 
weather.  To  identify  particularly  difficult  aspects  of 
these  missions,  CMC  performed  a  human  factors 
analysis  of  the  SAR  mission  in  very  poor  flight 
conditions  with  a  100-foot  ceiling  and  one  eighth  of  a 
nautical  mile  visibility  (1).  They  identified  several 
significant  issues,  two  of  which  are  particularly 
interesting:  first,  they  identified  mission  task 
elements  for  which  a  high  pilot  workload  is  necessary 
for  successful  completion;  and  second,  mission  task 
elements  that  are  impossible  to  perform  without 
enhanced  equipment. 

To  solve  these  problems,  an  enhanced  and  synthetic 
vision  system  was  proposed  to  alleviate  high 
workload  situations  and  augment  the  pilot’s 
situational  awareness  in  degraded  visual 
environments.  DND  commissioned  a  study  (2)  to 
outline  the  overall  requirements,  capabilities  and 
conceptual  design  for  the  augmented  ESVS  system. 

As  a  result  of  the  study  and  further  analysis,  the 
ESVS  system  was  targeted  toward  improving  SAR 
mission  performance  during  four  key  mission  phases 
in  degraded  visual  environments; 

•  first,  to  aid  the  pilot  in  descent  below  Minimum 
Descent  Altitude  in  a  search  area,  without 
prepared  navigation  systems  (i.e.  an  instrument 
landing  system  (ILS)  or  other  predefined 
instrument  approach  procedure); 

•  second,  during  the  search  for  the  crash  site, 
survivors,  and  wreckage  in  unknown  terrain; 

•  third,  after  the  crash  site  has  been  identified  and 
the  helicopter  is  preparing  to  rescue  the 
survivors,  to  aid  the  pilot  in  the  transition  to  and 
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Figure  1  ESVS  Display  Concept 


maintenance  of  hover  during  rescue  operations; 
and, 

•  finally,  to  help  the  pilot  depart  safely  from  the 
rescue  area. 

To  improve  performance  in  these  areas,  the  ESVS 
system  displays  an  augmented  visual  scene  to  the 
pilot  that  includes  three  separate  image  sources:  a 
synthetic  computer-generated  terrain  image;  an 
enhanced  visual  image  from  an  electro  optical  sensor; 
and  aircraft  instrument  symbology.  Figure  1  shows 
the  ESVS  display  concept.  The  system  provides  an 
immersive  environment  that  simulates  VFR 
conditions  using  a  visually  coupled  system  made  up 
of  a  helmet-mounted  display  system,  a  head  tracker 
and  a  slewable  camera  platform,  all  of  which  allows  a 
wide  variation  in  the  pilot’s  point  of  regard. 

The  synthetic  image  provides  the  pilot  with  a  wide 
field-of-view  terrain  display  that  can  be  used  to 
maintain  a  global  sense  of  position  and  orientation 
and  to  navigate  en-route  by  recognising  landmarks 
that  would  otherwise  be  hidden  by  poor  visibility. 
The  synthetic  image  is  generated  in  real  time  from 
aircraft  navigation  system  data  and  the  pilot’s  view 
point  based  on  head  orientation. 

The  enhanced  image  is  a  high  confidence  image  that 
can  be  used  during  manoeuvring  close  to  the  ground. 
As  the  pilot’s  head  turns,  the  head  tracker  reads  the 
pilot’s  head  position  and  sends  a  signal  to  the  sensor 
platform  telling  it  to  follow  the  pilot’s  head 
movement.  The  central  region  of  the  image 
comprises  the  enhanced  vision  system  output  fused 
with  the  synthetic  image. 


The  flight  symbology  provides  the  primary  flight 
references  necessary  for  the  pilot  to  fly  the  aircraft: 
airspeed,  altitude,  attitude,  power,  a  turn  co-ordinator, 
and  a  compass. 

But  why  are  both  enhanced  and  synthetic  images 
necessary?  In  short,  the  sum  utility  of  the  two  images 
is  greater  than  the  utility  of  either  one  by  itself 
because  each  image  has  its  own  advantages  and 
limitations. 

For  instance,  the  synthetic  image  is  always  available 
to  the  pilot  and  is  displayed  over  a  wide  field-of- 
view.  However,  it  does  not  include  all  objects  in  the 
viewing  region  and  the  accuracy  of  the  synthetic 
image  depends  on  the  accuracy  of  the  database  from 
which  it  is  created,  meaning  that  inaccuracies  in  the 
database  will  be  reproduced  in  the  synthetic  image. 
This  is  a  critical  shortfall  when  manoeuvring  close  to 
the  ground. 

Similarly,  the  enhanced  image  has  texture  and  culture 
detail  lacking  in  the  synthetic  image  giving  the  pilot 
higher  confidence  in  the  veracity  of  the  enhanced 
image.  On  the  downside,  the  enhanced  vision  sensor 
has  a  much  narrower  field-of-view  than  the  synthetic 
image.  By  fusing  both  images,  the  pilot  will  be  able 
to  use  the  best  aspects  of  each  image  overcoming  the 
limitations  of  each. 

ESVS  System  Architecture 

Figure  2  shows  the  ESVS  conceptual  architecture. 
Large  parts  of  the  subsystems  are  constructed  from 
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Figure  2  ESVS  Architecture  Block  Diagram 


commercial  off  the  shelf  items  (COTS)  with  custom 
software.  CAE  is  building  the  synthetic  image 
generator  (IG)  which  will  combine  a  map  database 
and  head  tracker  data  to  produce  a  synthetic  landform 
image  based  on  the  pilot’s  view  point.  The  synthetic 
image  and  the  enhanced  sensor  image  will  be  fused  in 
the  digital  image  processing  unit  (DIPU),  a  prototype 
of  which  has  been  built  by  CMC.  The  image  will 
then  be  displayed  to  the  pilot  using  a  helmet  mounted 
display  system  built  by  CAE. 

In  the  concepmal  ESVS  system,  the  image  generation 
system  will  produce  a  synthetic  terrain  image  using 
existing  terrain  map  databases  available  from 
Canadian  Government.  The  synthetic  image  will  also 
include  a  representation  of  non  database  objects 
detected  by  an  aircraft-mounted  range-finding  sensor. 
This  sensor  will  sweep  the  area  of  interest  to 
determine  the  actual  position  of  features  in  the  terrain 
database,  providing  the  ability  to  register  the 
synthetic  image  so  that  it  overlays  the  real  world. 
The  range  sensor  will  also  be  able  to  detect  feamres 
not  present  in  the  terrain  database,  and  provide  this 
information  to  the  image  generator  as  well.  The 
registration  and  detection  fimctions  provide  greater 
image  accuracy  and  flight  safety  and  are  among  the 
most  difficult  to  implement. 

The  ESVS  sensor  will  be  oriented  to  the  pilot’s  point 
of  regard.  As  with  the  synthetic  image,  the  enhanced 


image  will  not  be  sufficient  by  itself  to  provide  all  the 
visual  cues  necessary  to  control  the  aircraft  to  the 
pilot.  The  enhanced  image  will  contain  textural  and 
detail  cues  that  will  not  be  available  in  the  synthetic 
image.  The  synthetic  image  and  the  enhanced  sensor 
image  will  be  fused  in  the  digital  image  processing 
unit.  Various  image  fusion  techniques  will  be  tested 
to  determine  the  best  way  to  present  the  combined 
enhanced  and  synthetic  image  to  the  pilot.  It  is 
expected  that  the  pilot  will  have  control  over  the 
fusion  process  and  will  be  able  to  display  the 
unprocessed  sensor  image  if  required. 

The  sensors  being  considered  for  the  enhanced  image 
include  infrared,  low  light  level  TV  and  small 
wavelength  radar.  The  optical  elements  for  the 
infrared  or  simulated  low  light  level  TV  will  be 
chosen  with  an  appropriate  field-of-view  that 
matches  the  pixel  size  of  the  area  covered  by  the 
enhanced  image  within  the  synthetic  image.  This 
eases  the  processing  load  required  to  fuse  the 
enhanced  and  synthetic  images. 

Head  position  will  be  provided  by  a  head  tracking 
system.  The  head  tracker  information  will  be  used  to 
drive  the  sensor  platform  and  provide  view  point 
orientation  for  the  image  generation  system  and  the 
symbology  generator. 
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Figure  3  NRC  Bell  205 


Experimental  System 

NRC  will  be  integrating  and  testing  a  prototype 
demonstration  ESVS  to  be  fielded  in  the  year  2000. 
The  functionality  of  the  demonstration  system  is 
reduced  from  the  conceptual  ESVS  because  the  range 
finding  system  has  been  deleted. 

The  demonstration  system  will  be  installed  in  the 
Flight  Research  Bell  205  Airborne  Simulator  shown 
in  Figure  3.  The  ESVS  image  will  be  displayed  on 
the  SPIRIT  II  airworthy  helmet  mounted  display 
system  produced  by  CAE.  A  PoUiemus  3Space 
magnetic  head  tracker  will  provide  head  position 
information. 

The  SPIRIT  HMD  is  a  binocular  high  resolution  wide 
field-of-view  display.  The  display  is  55°x40°  in  each 
eye,  with  a  30°  binocular  overlay  giving  in  a  total 
field-of-view  of  80°.  The  display  consists  of  a 
transmissive  ferro-electric  liquid  crystal  display 
(FELCD)  backlit  by  a  laser  light  source,  and  has  a 
resolution  of  1280x1024  pixels. 

The  airborne  simulator  has  a  full  authority  fly-by¬ 
wire  control  system  with  specially  designed  dual¬ 
mode  electro-hydraulic  actuators  to  allow  the  system 
to  be  controlled  either  mechanically  by  the  safety 
pilot  or  electrically  by  the  evaluation  pilot.  The  use 
of  the  fly-by- wire  system  allows  the  augmentation  of 
the  basic  helicopter  stability  to  offset  the  degraded 
visual  cues  provided  by  the  ESVS  system.  The 
aircraft  navigation  instrumentation  includes  both 
inertial  navigation  information  and  differential  GPS, 


used  to  compute  aircraft  position  with  a  real-time 
accuracy  of  one  meter. 

During  any  test  at  NRC,  two  pilots  operate  the 
aircraft.  The  evaluation  pilot  assesses  the  ESVS 
while  the  safety  pilot  monitors  flight  safety.  If  the 
evaluation  pilot  encounters  difficulty  controlling  the 
aircraft  or  interpreting  the  displayed  visual  scene,  the 
safety  pilot  can  take  over  control  of  the  aircraft.  The 
aircraft  also  contains  a  video  and  audio  recording 
system  that  monitors  the  evaluation  pilot,  the  sensor 
and  symbology  visuals  and  the  intercom  channels  to 
allow  analysis  of  any  problems  reported  by  the  pilots. 


ESVS  Research  Program 

A  comprehensive  research  program  has  been 
developed  to  understand  the  limitations  of  the 
demonstration  ESVS  hardware,  to  promote  a 
fundamental  understanding  of  human-machine 
cockpit  interfaces,  and  to  gain  experience  in  the 
integration  of  sophisticated  cockpit  technology.  The 
research  is  being  conducted  at  four  facilities:  CAE’s 
offices  in  Montreal,  Quebec;  the  CMC  Systems 
Integration  Facility  in  Kanata,  Ontario;  the  ground- 
based,  moving  base  flight  simulator  at  UTIAS  in 
Toronto;  and  at  NRC’s  Flight  Research  Laboratory  in 
Ottawa,  Ontario. 

An  effective  ESVS  will  be  based  on  a  solid 
understanding  of  human  performance  and 
technological  limitations.  The  ESVS  team  has 
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identified  a  list  of  high  priority  research  topics, 
including; 

•  the  determination  of  an  effective  field-of-view 
for  the  required  tasks; 

•  an  understanding  of  the  effect  of  limited  sensor 
platform  roll  motion  on  pilot  behaviour; 

•  quantification  of  the  effects  of  system  latencies 
on  pilot  performance; 

•  the  design  of  symbology  sets  to  overcome  ESVS 
system  deficiencies; 

•  selection  of  an  appropriate  image  fusion 
technique; 

•  analysis  of  the  trade-off  between  binocular  and 
biocular  imagery;  and 

•  evaluation  of  scene  content  issues. 

NRC  has  already  completed  a  number  of 
investigations  into  these  topics.  Each  research 
project  is  described  below,  along  with  the  status  of 
the  study. 

Experiments  are  ongoing  into  the  effectiveness  of 
HMD  systems  in  helicopter  operations  using  a  Fibre 
Optic  Helmet  Mounted  Display  (FOHMD)  which 
was  installed  in  NRC’s  Bell  205  helicopter  in  1996 
(4).  The  FOHMD  system  has  a  larger  FOV  than  the 
SPIRIT  system  (total  FOV  of  105°  with  a  25° 
binocular  overlap  region),  but  has  a  lower  resolution 
image  (754x485  pixels).  Baseline  performance  of  the 
FOHMD  system,  reported  in  (5),  shows  handling 
qualities  as  a  solid  Level  2  system  in  degraded  visual 
environments  when  using  a  rated  damped  control 
system. 

Field-of-View 

A  study  was  conducted  to  investigate  the  effect  of 
field-of-view  and  amount  of  binocular  overlap  on 
pilot  performance  (3).  The  tests  were  carried  out  in 
NRC’s  Bell  205  and  206  helicopters  using  a  set  of 
goggles  with  cut-outs  that  were  set  up  to  restrict  the 
pilot’s  vision.  The  experiments  showed  that  a 
binocular  overlap  region  is  necessary  for 
manoeuvring  close  to  the  ground,  landing  and  other 
nap  of  the  earth  operations.  While  larger  FOV  is 
better,  improvements  in  handling  qualities  are 
marginal  beyond  a  100°  FOV  and  Level  1  handling 
qualities  are  maintained  with  at  least  an  80°  FOV. 
The  results  of  these  tests  indicate  that  the  field-of- 
view  and  binocular  overlap  provided  by  the  SPIRIT 
II  HMD  system  are  acceptable  to  perform  the  ESVS 
mission. 

Roll  Compensation 

The  present  NRC  sensor  platform  has  yaw,  pitch  and 
roll  motion  capability,  unlike  most  current  sensor 


platforms  that  do  not  have  full  three  degree  of  motion 
systems.  Due  to  the  complexity  and  cost,  production 
systems  are  usually  restricted  to  yaw  and  pitch 
motion.  The  current  series  of  NRC  tests  is  looking  at 
the  effects  of  restricting  the  amount  of  camera 
platform  roll  on  aircraft  handling.  Preliminary  results 
of  the  study  indicate  that  some  platform  roll  is 
necessary  for  proper  aircraft  operation  when  using 
the  fully  immersive  FOHMD  system. 

System  Delays 

The  ESVS  will  require  a  significant  amount  of 
graphics  processing  of  the  displayed  image  which  can 
potentially  introduce  time  delays  into  the  visual 
display  system.  Pilot  co-ordination  in  handling 
aircraft  can  be  degraded  by  large  delays  in  image 
presentation  on  the  HMD.  NRC  is  conducting  an 
investigation  to  study  the  effect  of  time  delay  on 
aircraft  handling  qualities.  This  information  will  be 
used  to  determine  the  maximum  allowable  processing 
delay  for  the  graphics  system. 

Symbology 

The  current  NRC  HMD  symbology  has  evolved  over 
the  course  of  two  years  of  testing  at  NRC  and 
UTIAS.  A  systematic  flight  test  program  for  HMD 
symbology  is  planned  for  the  next  year  to  look  at  the 
minimal  set  of  instruments  required  for  navigation 
and  flight  maintenance.  Questions  to  be  answered 
include  the  acceptable  level  of  clutter,  the  required 
compatibility  with  other  sources  of  information  in  the 
cockpit,  the  optimal  format  (including  different 
configurations  for  different  mission  phases),  and 
integration  issues. 

Image  Fusion  Techniques 

As  mentioned  earlier,  the  enhanced  image  will  be 
fused  into  the  synthetic  image  and  displayed  to  the 
pilot.  There  are  a  number  of  issues  that  need  to  be 
investigated  regarding  this  image  fusion  such  as  how 
close  the  registration  of  the  enhanced  and  synthetic 
image  must  be  before  the  image  causes  problems 
with  the  pilot’s  ability  to  handle  the  aircraft,  and 
under  what  conditions  the  pilot  may  want  to  control 
the  fusion  algorithm.  The  results  of  tests  at  the 
UTIAS  simulator  will  be  evaluated  in-flight  when  the 
systems  and  results  are  available. 

Binocular  and  Biocular  Trade-offs 

The  use  of  two  optical  sensors  on  the  camera 
platform  adds  to  the  complexity  and  weight  of  the 
fielded  system.  However,  two  sensors  are  necessary 
to  produce  a  true  binocular  image.  The  CMC  DIPU 
has  the  capability  to  use  one  camera  image  to 
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produce  biocular  images  for  display  on  the  HMD. 
NRC  plans  to  run  an  experiment  to  investigate  the 
differences  in  handling  qualities  when  using  a 
biocular  image  versus  a  true  binocular  image. 

Scene  Content 

Simulator  studies  have  been  carried  out  at  the  UTIAS 
simulator  using  the  map  database  of  the  selected  test 
site  area  to  investigate  the  graphics  image 
requirements  for  the  synthetic  image  system.  NRC 
will  be  looking  at  scene  content  issues  based  on  the 
finding  from  the  UTIAS  studies.  The  issues  to  be 
investigated  include  image  texture,  level  of  cultural 
detail  and  problems  with  producing  the  graphics 
image  from  the  terrain  database.  The  problems 
include  such  anomalies  as  rivers  which  flow  up  hill 
and  lakes  with  contours  due  to  inaccuracies  in  the 
geographical  database. 

Conclusions 

Search  and  Rescue  operations  are  important  in  any 
country,  but  with  the  size  and  remoteness  of  large 
areas  of  Canada,  the  capability  of  carrying  out  SAR 
missions  in  day  and  night,  all  weather  conditions  is  of 
paramount  importance.  The  enhanced  and  synthetic 
vision  system  described  in  this  paper  will  help 
alleviate  many  problems  facing  SAR  pilots  flying 
aircraft  equipped  with  present  day  technology.  The 
ESVS  team  is  working  toward  preparing  a 
demonstration  system  by  the  year  2000  and  in 
parallel,  are  conducting  a  research  program  to 
understand  the  human  factors  issues  associated  with 
the  ESVS  system. 
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1  SUMMARY 

Inappropriate  automation  is  considered  to  be  a  major  reason  for 
deficiencies  on  interaction  between  pilot  crew  and  aircraft 
systems.  The  lack  of  situation  awareness  is  pointed  out  to  be  a 
crucial  cause  of  pilot  failure.  Because  of  this,  cockpit  assistant 
systems  are  being  developed  in  support  of  human-centered 
automation.  CAMA  assists  military  crews  during  transport  mis¬ 
sions. 

This  paper  consists  of  three  main  parts,  briefly  describing  the 
functional  prototype  of  CAMA,  the  experimental  means  taken 
in  order  to  evaluate  the  integrated  system,  and  the  comprehen¬ 
sive  results  of  two  flight  simulator  campaigns. 

Firstly,  a  general  survey  is  given  on  human  factors  related 
problems  in  this  particular  domain.  Their  influence  on  the 
principals  of  cockpit  crew  assistance  will  be  shown  and  a  brief 
circumscription  of  CAMA's  main  functionalties  follows. 

The  description  of  the  simulator  facilities  for  experimentation 
includes  the  visual  system,  the  available  flight  controls  and  the 
means  for  interaction  between  the  pilot  and  CAMA.  The  ex¬ 
perimental  scenario  and  tasks  are  pointed  out. 

To  get  an  estimation  on  the  pilot’s  overall  acceptance  of  the  ap¬ 
proach  and  the  benefits  of  the  CAMA-system,  thorough 
evaluations  were  conducted. 

2  INTRODUCTION 

Military  transport  pilots  are  confronted  with  both,  flying  under 
IFR  conditions  in  high  density  airspace  as  well  as  performing 
tactical  multi-threat  missions.  Due  to  the  variety  of  tasks  that 
have  to  be  handled,  crews  suffer  from  a  high  workload. 
Situations  inevitably  arise  where  the  crew  members  are  over¬ 
taxed  regarding  their  limited  mental  information-processing 
resources,  and  thereby  act  erroneously.  Because  of  this,  situa¬ 
tions  may  develop  which  jeopardise  the  mission  success.  The 
high  crew  workload  is  caused  by  the  variety  of  different  tasks, 
an  insufficiently  adapted  crew  interface,  and  the  poor  avail¬ 
ability  of  information  relevant  to  the  situation  and  task  in  the 
cockpit.  Closely  related  to  this  is  the  problem  of  situation 
awareness.  Situation  awareness  has  been  achieved  when  the 
pilot  has  all  relevant  information  which  is  required  to  resolve 
the  most  urgent  task  at  his  disposal  [7].  This  includes  the  ob¬ 
jectively  correct  awareness,  which  is  actually  the  most  urgent 
task.  Situation  awareness  is  therefore  the  result  of  a  continuous 
process  of  situation  assessment  [18]. 

Human-centred  automation  [1]  appears  to  be  a  promising  ap¬ 
proach  to  the  design  of  future  cockpit  avionics  functions.  The 
starting  point  is  the  analysis  of  the  crew’s  tasks.  These  tasks 
comprise  information  gathering,  situation  interpretation  and 
analysis  including  the  ownship  situation  as  well  as  tasks  relating 
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to  in-flight  mission  planning  and  tactical  low-level  flight 
trajectory  computation,  and  finally  flight  guidance  and  naviga¬ 
tion  tasks. 

A  technical  system  which  is  meant  to  be  an  effective  crew 
assistant  system  for  the  transport  mission  management  should 
cover  at  least  the  above-mentioned  chain  of  functions.  Given 
this  basis  of  crew  assistance  it  ought  to  be  taken  into  consid¬ 
eration  that  the  assistant  system  should  not  replace  the  crew  for 
certain  tasks  in  terms  of  automation.  The  machine  should  be 
able  to  perform  the  tasks  in  parallel  with  the  human  operator 
and  in  co-operative  function  allocation  [7].  Like  the  human 
operator,  the  machine  part  of  such  a  man-machine  system  also 
needs  to  have  the  information  relevant  to  the  situation  (task)  at 
its  disposal. 

Incorporating  these  guidelines,  the  Crew  Assistant  Military 
Aircraft  (CAMA)  was  developed.  The  subsequent  sections  deal 
with  evaluation  in  simulator  trials.  Details  will  be  given  on  the 
experimental  setup,  the  investigated  scenarios  and  the  found 
results. 

3  CREW  ASSISTANT  MILITARY  AIRCRAFT 

The  Crew  Assistant  Military  Aircraft  (CAMA)  is  a  knowledge- 
based  cognitive  assistant  system  under  development  in  close 
cooperation  between  the  partners  ESG,  the  University  of  the 
German  Armed  Forces,  the  German  Aerospace  Research 
Establishment  (DLR),  and  DASA  since  1995.  CAMA  and  its 
philosophy  have  been  described  in  various  other  papers  such  as 
[14][16][17],  so  that  there  is  no  need  to  go  into  detail  too  much, 
here.  However,  the  following  paragraphs  will  give  a  bief  insight 
into  the  functional  modues  of  CAMA.  Figure  1  shows  the 
functional  structure  of  CAMA. 

The  interface  between  CAMA  and  the  crew  is  controlled  by  the 
module  Dialogue  Manager  (DM).  Speech  output  is  suitable  to 
focus  the  pilot's  attention  on  important  aspects.  More  complex 
information  is  transmitted  using  graphical  displays  [13]. 
Information  input  is  realised  utilising  speech  recognition  to  a 
large  extent  [3][4]. 

Various  modules  provide  crucial  information  on  health  status  of 
aircraft  systems  (SI),  environmental  aspects  (El),  and  the  flight 
progress. 

The  Tactical  Situation  Interpreter  (TSI)  calculates  the  local 
threat  distribution  along  the  mission  plan.  Thereby,  CAMA  is 
able  to  perform  the  conflict  detection  with  respect  to  local 
changes  in  the  tactical  situation  [10]. 

In  order  to  ensure  situation  awareness  for  the  machine-system, 
all  information  produced  by  the  modules  is  assembled  in  a 
Central  Situation  Representation  (CSR)  and  this  way  provides  a 
complete  dynamic  database  of  the  current  situation.  This  can  be 
seen  analoguously  to  the  pilot's  own  mental  representation  of 
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16-2 


the  actual  situation.  It  ensures  that  the  system  is  working  on  the 
same  assumptions  as  the  crew,  thus  making  it  different  from 
existing  cockpit  functions  which  make  use  of  only  a  restricted 
amount  of  relevant  knowledge. 

On  the  basis  of  situation  knowledge,  possible  conflicts  ahead 
can  be  identified  (e.g  threats,  weather)  by  the  Flight  Situation 
and  Threat  Interpreter  (FTI).  Here  the  impact  on  the  current 
flight  is  assessed,  conflicts  are  detected,  and  resolution  activities 
are  initiated. 

The  Misson  Planner  (MP)  generates  a  complete  3D/4D  mission 
plan  [8]  either  on  demand  by  the  crew  or  autonomously,  in  case 
the  crew  not  having  the  resources  to  interact.  It  is  important  to 
mention  that  the  planning  process  again  takes  all  aspects  of  the 
situation  (provided  by  the  Central  Situation  Representation) 
into  consideration  [16]. 

If  the  penetration  of  hostile  area  is  inevitable  necessary,  the  Low 
Altitude  Planner  (LAP)  is  activated  to  achieve  a  threat  minimi¬ 
sing  low-level  routing  taking  the  output  of  the  aforementioned 
modules  into  consideration.  It  allows  the  incorporation  of 
corridors  and  tactical  delivery  procedures,  such  as  a  payload 
drop  procedure.  Both,  low-level  as  well  as  IFR-routing  are 
combined  to  on  homogenious  flight  plan. 

The  modules  Pilot  Behaviour  Interpreter  (PBI)  as  well  as  Pilot 
Intent  and  Error  Recognition  (PIER)  serve  mainly  for 
monitoring  of  pilot  behaviour,  which  shall  be  described  in  more 
detail  in  [14]  and  [15]. 

Dynamic  external  data  like  air  traffic  control  and  C&C  instruc¬ 
tions  as  well  as  environmental  information  are  gathered  via  an 
external  communication  interface. 

Static  databases  comprise  information  like  navigational,  terrain 
and  feature  data. 


4  SIMULATOR  TRIALS 

After  going  through  a  module  intergration  phase  of  several 
months,  a  scientific  approach  was  chosen  in  order  to  evaluate 
CAMA.  The  following  sections  will  describe  the  experimental 
design. 

4.1  Apparatus 

CAMA  has  been  integrated  and  tested  in  the  research  flight 
simulator  located  at  the  University  of  the  German  Armed  Forces 
at  Munich  (see  Figure  2  for  a  schematic  view). 


Figure  2;  Experimental  Simulator  Configuration 


Flight  simulation  and  visual  simulation  run  on  Silicon  Graphics 
workstations.  The  dynamic  model  is  derived  from  the  ATTAS 
experimental  aircraft  of  the  DLR.  A  wide  field-of-view  visual 
simulation  system  provides  visual  cues  to  perform  low-level 
flight  missions.  The  database  was  derived  from  DMA  DTED 
and  DFAD  [19]  data,  giving  information  on  terrain  elevation  as 
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well  as  on  topolgrahical  features  (airports,  populated  areas, 
forrests,  rivers,  powerlines,  etc.)  in  order  to  enable  visual 
navigation, 

the  pilot’s  station  is  a  generic  glass  cockpit  equipped  with 
monitors  presenting  the  CAMA-Displays: 

•  Primary  Flight  Display:  It  incorporates  two  basic  display 
types.  A  standard  ADI  display  in  order  to  support  the  IFR 
portions  of  the  mission  and  a  three-dimensional  perspec¬ 
tive  format  (Figure  3)  enhancing  low-level  flight  guidance 
in  particular  under  restricted  visual  conditions.  The  latter 
not  only  provides  information  to  the  surrounding  terrain 
and  feature  data,  but  also  basic  aircraft  parameters.  It  fur¬ 
ther  depicts  the  planned  trajectory,  generated  by  CAMA's 
low-level  planning  module  [6]  and  the  evasive  maneuvres, 
generated  by  CAMA's  terrain  interpreter.  Both  are 
displayed  as  a  three-dimensional  flight  guidance  tunnel. 
Additionally,  the  expected  behaviour  parameters  generated 
by  the  pilot  model  are  displayed  and  respective  deviations 
in  the  actual  behaviour  of  the  pilot  are  indicated. 


Figure  3:  Three-Dimensional  Flight  Guidance  Display 


•  Interactive  navigation  display:  This  multimodal  display  [3] 
provides  a  variety  of  functions.  Most  important,  it  supports 
the  pilot  during  lateral  navigation,  showing  his  position  in 
reference  to  several  situational  elements.  Depending  on  the 
actual  task  during  flight,  navaids,  airports,  weather, 
converging  traffic  (TCAS-symbology),  flight  plan,  SAM 
locations  and  ACO  information  are  culled  and  displayed. 
Terrain  information  and  the  calculated  threat  map  [11]  are 
underlayed.  Resulting  from  CAMA's  internal  situation 
assessment,  certain  symbols  are  highlighted  or  blinking  in 
order  to  attract  the  pilot's  attention  (e.g.  new  SAMs, 
blocked  corridors]  etc.).  In  parallel  the  relevant  speech 
output  is  displayed  on  the  upper  edge  (see  Figure  4).  By 
touching  the  symbols  on  the  map,  the  pilot  can  enquire 
more  information  concerning  these  elements  and/or  subject 
them  to  further  processing.  For  example,  navaids  can  be 
picked  directly  via  touch  input,  new  corridors  can  be 
selected  for  a  replanning,  etc. 

•  The  Secondary  Dislplay  supplements  the  Navigation  Dis¬ 
play  by  presenting  a  flight  log,  ATIS  information,  and  ap¬ 
proach  charts  on  the  pilot's  request. 

Primary  flight  control  is  performed  by  a  sidestick,  an  Airbus- 
type  Flight  Control  Unit,  throttle,  etc. 


Figure  4:  Interactive  Navigation  Display 

Another  important  media  for  information  exchange  between  the 
pilot  and  is  speech  recognition  /  synthesis  [4],  Using  a  SUN 
workstation  based  speech  recognition  system  the  pilot  is  able  to 
communicate  the  system  on  a  “natural  speaking”  level.  In  order 
to  enhance  recognition  probability,  situation  adapted  syntaxes 
are  generated  and  used.  The  following  list  shows  the  main 
functionalities  that  can  be  requested  by  the  pilot  via  speech 
input. 


Command 

Function 

"request  flight  plan  to ... 

(arpt  id) 

"request  flight  plan  via  ...  (wptt  id) 
"activate  flightplan ...  (proposal 
number)" 

automatic  re-planning 

“ climb/Descent ...  feet/flightlevel" 

"increase/reduce ...  knots” 

"turn  left/right ...  (hdg)" 

"proceed  to ...  (wpt)” 

autopilot  settings 

"display  heading  up/ 
north  up/ 
rose  mode/ 

"zoom  ...  miles/ 

mini/maxi " 

display  configuration 

"select ...  (nav  id)  on  vorI/vor2/adf  " 

"select  course" 

“select  ils” 

navaid  configuration 

“request  alternates" 

“request  ATIS  of ...  (arpt  id)" 

“request  distance  to ... 

(arpt  id/ navaid  id/ ...)" 
request  approach/departure 
briefing/chart  ” 

supplementary  information 

Table  1:  Speech  Syntax 


In  addition  to  the  information  given  on  the  displays,  CAMA 
issues  important  messages  by  voice  output  to  the  pilot.  Re¬ 
garding  the  importance  of  the  message  different  voice  levels  are 
chosen. 

The  simulator  includes  a  operator  console,  allowing  interference 
during  mission  progress.  Researchers  can 
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•  control  situational  elements  (weather,  other  aircraft,  etc.), 

•  dispatch  ATC  /  C&C  commands  via  datalink, 

•  control  data  acquisition  and  eye  movement  measurement. 

4.2  Subjects 

The  subjects  were  ten  German  Air  Force  transport  pilots  (Air- 
lifter  Wing  61,  Landsberg)  between  the  age  of  29  and  43  (mean 
age  38.4  years).  The  subjects  were  well  experienced  (combat 
ready)  and  had  between  800  and  5300  total  flying  hours  on  the 
C-160  Transall  (mean  flying  hours  2550).  Figure  5  shows  a 
pilot  interacting  with  the  CAMA  system  during  the 
experimental  flight. 


Figure  5:  Pilot  in  Experimantal  Cockpit 


4.3  Scenario  and  Tasks 

Prior  to  the  evaluation  phase,  time  was  given  for  familiarisation 
and  training  on  the  system.  Emphasis  was  put  on 

•  basic  aircraft  handling, 

•  aircraft  guidance  in  low-level  flight  using  the  three-dimen¬ 
sional  flight  guidance  display, 

•  use  of  the  autopilot, 

•  use  of  the  interactive  navigation  /  information  displays, 

•  use  of  speech  recognition,  and 

•  interpretation  of  speech  output. 

The  tasks  to  be  performed  during  the  evaluation  phase  were 
described  by  a  full  scale  air  transport  mission,  consisting  of 
segments  under  IFR  conditions  followed  by  tactical  low-level 
flight. 

The  IFR  scenario  incorporated 

•  adverse  weather  conditions, 

•  high  density  airspace, 

•  varying  availability  of  landing  sites,  and 

•  extensive  ATC  communication. 

The  tactical  scenario  was  characterised  as  a  dynamic  multi¬ 
threat  theatre  due  to  the  presence  of 

•  surface-to-air  missile  sites, 
and  tactical  constraints  such  as 

•  air  tasking  order  (ATO),  and 

•  airspace  coordination  orders  (ACO). 

Before  take-off,  the  subjects  received  a  standard  mission 
briefing  using  conventional  maps,  but  also  already  CAMA's 
displays. 


Take-off  was  performed  at  a  controlled  airport.  During  the  IFR 
transit  flight  into  the  tactical  operation  area  the  subjects  had  to 
perform  several  planning  and  re-planning  tasks.  These  were 
induced  by  thunderstorm  areas  and  ATC  requests. 

After  passing  the  entrance  corridor  to  the  hostile  area  the 
mission  continued  in  low-level  flight,  avoiding  known  threats 
where  possible,  minimising  the  exposure  to  unknown  threats 
and  keeping  clear  of  the  terrain.  Next,  a  tactical  drop  procedure 
had  to  be  performed  while  being  loaded  by  additional  planning 
tasks  concerning  the  low-level  route  ahead. 

Having  left  the  exit  corridor  the  pilot  resumed  the  IFR  flight. 
Converging  traffic  in  terminal  areas  and  closed  airports  forced 
the  pilot  to  make  various  short-term  and  medium-term  deci¬ 
sions. 

4.4  Experimental  Plan 

Each  subject  had  to  perform  the  mission  3  times,  once  on  each 
of  the  three  experimental  days.  The  simulator  trials  took  place 
in  autumn  1997  and  spring  1998.  Table  2  gives  the 
experimental  time  schedule  for  the  first  evaluation  period.  In  the 
second  experimental  period  the  schedule  was  repeated,  but  only 
the  second  flight  evaluated. 


Experiment 

Duration 

Introduction  to  simulator 

30  rain 

Simulator  familiarisation  flight 

45  min 

CAMA  familiarisation  flight 

45  min 

1"  experimental  CAMA  mission 

60  min 

Debriefing  and  questionnaires 

120  min 

Pilot  behaviour  model  tuning  flights 

180  min 

2"**  experimental  CAMA  mission 

60  min 

Debriefing  and  questionnaires 

1 20  min 

Table  2:  Experimental  Time  Schedule 


5  RESULTS 

In  this  chapter  the  results  of  an  extensive  experimental 
evaluation  of  the  CAMA  functional  prototype  are  summarised. 
The  main  focus  is  on  the  subjective  ratings  given  by  the 
experimental  pilots.  A  few  statements  concerning  objective 
evaluation  criteria  are  added  afterwards. 

5.1  Subjective  ratings 

In  order  to  assess  the  pilot’s  overall  acceptance  of  the  approach 
and  the  benefits  offered  by  the  cockpit  assistant  system,  a 
debriefing  session  concluded  the  experiments  as  described 
above.  The  evaluation  results  presented  here  give  an  overview 
over  the  subjective  ratings  given  by  the  experimental  pilots 
concerning  the  tactical  cockpit  assistant  functions,  as  well  as 
concerning  the  IFR-operations  related  cockpit  assistance 
provided  by  CAMA. 

All  ratings  were  given  within  a  range  from  1  through  7,  where  1 
represents  the  best  evaluation  and  7  marks  the  negative  end  of 
the  scale.  The  diamonds  mark  the  median  value  of  the  ratings 
given  after  the  first  (1)  and  second  (2)  experiment  of  the  first 
campaign.  (3)  stands  for  the  flight  performed  during  the  second 
campaign. 

First  of  all,  it  was  necessary  to  ensure  that  the  simulation  envi¬ 
ronment  was  adequate  for  performing  the  required  tasks.  There¬ 
fore,  several  aspects  of  the  simulation  which  are  relevant  to  the 
performance  of  tactical  flight  missions  were  evaluated. 
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Figure  6  displays  some  summarised  results  of  the  evaluation 
with  respect  to  the  flight  simulation  environment. 


(a)  Flight  Simulation  System  Handling 


-  !  • 

1  ' 

1  :  '1 

2 

3  4 

7 

(b)  Visual  Simulation  appropriate  for  Low-level  Flight 

#  i 

'1  1 

I'  V/V/W  =  Iff  j  I _ L_^ _ I 

1  2  3  4  7 


Figure  6:  Evaluation  of  flight  simulation 

Here  the  evaluation  of  the  flight  simulation  handling  (a)  com¬ 
prises  several  parameters  such  as  manual  sidestick  control, 
FCU,  and  RMU  operations  during  IFR  and  tactical  flight.  It  can 
be  stated  that  the  acceptance  of  the  experimental  platform 
slightly  improved  over  the  course  of  three  flights,  and  is  cer¬ 
tainly  satisfatory  as  prerequisite  for  the  CAMA  evaluation. 

In  order  to  gain  an  overall  impression  of  CAMA's  performance, 
the  following  statements  were  evaluated.  Figure  7  shows  the 
rating  results,  again  measured  on  a  scale  from  1  (strong 
agreement)  to  7  (strong  disagreement). 


(a)  '[always  understood  CAMA's  Actions.’ 


(b)  7  was  (made)  aware  of  my  own  faults. ' 

^  <3>i  <^>  i  ■  I  j-> . j  I  L . r 

1  2  3  4  7 

Figure  7:  Evaluation  of  CAMA  philosophy 

The  ratings  show  that  the  pilots  were  able  to  understand  the 
intents  and  actions  of  CAMA  (a)  only  after  the  short  familiari¬ 
sation  of  one  flight.  A  significant  training  effect  was  observed 
from  the  first  flight  with  CAMA  to  the  second  campaign,  as  the 
ratings  are  improving  noticably.  It  is  significant  that  it  does  not 
only  seem  to  be  an  effect  of  familiarity  with  the  simulation 
facility  and  displays,  but  an  apparent  increase  in  understanding 
the  aims  followed  by  CAMA  (see  Figure  7  (a)). 

The  methodological  approach  to  the  subjective  evaluation  of 
CAMA  followed  a  certain  scheme  of  questions,  which  has  been 
derived  from  the  considerations  mntioned  below.  [9]  proposes 
the  following  aspects  for  the  evaluation  of  man-machine- 
systems: 

Compatibility:  Is  the  system  adapted  to  the  operators  capabili¬ 
ties  and  limitations? 

Intelligibility:  Does  the  organisation  and  the  contents  of  the 
man-machine-dialogue  provide  a  meaningful  communi¬ 
cation? 

Effectiveness:  To  what  extent  does  the  system  enhance  task- 
performace? 

According  to  [5],  meeting  these  three  criteria  is  a  prerequisite  to 
reach  a  high  degree  of  acceptance  for  the  man-machine-system. 
Another  important  aspect  for  the  evaluation  of  a  crew  assitant 
system  is  the  term  of  situation  awareness.  [2]  defines  situation 
awareness  as  follows: 


“Situation  awareness  is  the  perception  of  the  elements  in  the 
environment  ...  ,  the  comprehension  of  their  meaning,  and  the 
projection  of  their  status  in  the  near  future.” 

Therefore,  the  questionnaires  were  structured  according  to  the 
following  aspects: 

•  In  order  to  evaluate  the  situational  awareness  aspects,  the 
considered  situation  space  was  classified  and  structured  into- 
classes  and  sub-classes  of  relevant  situational  elements. 
Again,  with  regard  to  these  classes  the  pilots  had  to  indicate 
whether  they  had  all  the  situational  element-related  infor¬ 
mation  at  their  disposal  whenever  needed. 

•  The  effectiveness  and  the  benefit  provided  by  the  functions 
were  evaluated  by  listing  all  relevant  tasks  and  sub-tasks 
throughout  the  mission.  The  subjects  had  to  comment  on  the 
quality  of  assistance  provided  by  CAMA  with  respect  to 
these  tasks. 

•  To  evaluate  the  degree  of  acceptance,  the  pilots  had  to  refer 
to  a  list  of  statements  characterising  the  system  behaviour 
and  handling  features. 

Figure  8  depicts  the  situational  awareness  ratings  with  respect 
to  a  collection  of  situation  elements.  The  diamonds  on  the  bars 
show  the  median  values  of  the  pilots  ratings  evaluating  their 
subjective  sensation  of  situational  awareness.  It  should  be  kept 
in  mind  that  the  Figure  only  depicts  the  positive  half  of  the 
ranking  scale.  Again,  (1)  and  (2)  indicate  the  results  of  the  first 
experimental  campaign  and  (3)  stands  for  the  second  campaign 
with  CAMA. 

(a)  A  ircraft  A  ttitude 
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(b)  Aircraft  Position 

c>  \  i  M  I  I  I-  I 
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(c)  Terrain  Relief 
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(d)  Threat  along  Low-level  Flight  Trajectory 

^-  i4>  l  I  I  I  -  I  I  I 
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(e)  Re-planning  needs 
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Figure  8:  Evaluation  of  situation  awareness 

Figure  8  makes  clear  that  the  situation  awareness  ratings 
concerning  the  tactical  situation  elements  remained  unchanged 
over  the  two-day  experimentation  at  a  very  high  level.  On  the 
other  hand,  the  pilots  situation  awareness  on  primary  flight 
parameters  such  as  speed  or  altitude  is  certainly  more  crucial 
and  has  therefore  been  evaluated  more  critically. 

The  following  collection  of  ratings  shown  in  Figure  9  depicts 
the  pilot’s  rating  on  the  quality  of  assistance  provided  by  the 
CAMA  functions. 
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{a)  Comply  with  Mission  Constraints  (ACO/ATO) 


2  3 

(b)  Assess  Threat  Efficiency 


i,  I 


1  2 

(cj  Perform  IFR  Planning 


not  neccessary 
useless 
nonsense 
inappropriate 
too  often 
too  strong 


1  2  3 

(d)  Plan  Low-Level  Flight  Route 


2  3  4 

(e)  Update  Flight  Plan  due  to  ATC  Clearance  / Order 


Figure  10;  Evaluation  of  advice  and  warning  philosophy 

In  Figure  10  it  becomes  evident  that  CAMA’s  concept  of  giving 
advice  and  warning  is  very  well  accepted  by  the  pilots. 
Furthermore,  the  philosophy  appears  to  be  well  balanced  in 
terms  of  warning  frequency/sensitiveness  and  intensity. 

The  next  selection  of  experimental  evaluations  is  concerning  the 
subjective  measurement  of  the  degree  of  acceptance.  The 
acceptance  was  determined  by  evaluating  the  behaviour  of  the 
assistant  system,  as  well  as  its  handling  qualities,  as  indicated  in 
Figure  1 1  and  Figure  12. 


(g)  Pei  form  a  Tactical  Drop  Procedure 


Like  Figure  7  (a).  Figure  1 1  indicating  the  subjective  easyness 
of  the  CAMA  operations  shows  a  strong  training  effect.  The 
handling  of  CAMA  became  much  easier  during  the  course  of 
the  experiments. 


Figure  9:  Evaluation  of  assistance  quality 

Figure  9  (c)  and  (d)  clearly  prove  that  the  automatic  flight 
planning  functions  (for  IFR  and  low-level)  are  well  established 
in  the  context  of  the  CAMA  functions.  Their  performance  is 
well  appreciated  by  the  pilots.  The  low-level  flight  guidance 
and  navigation  assistance  (f)  provided  mainly  by  the  various 
features  of  the  three-dimensional  flight  guidance  display  and  the 
navigation  display  is  regarded  as  very  powerful.  The  effec¬ 
tiveness  of  the  tactical  assistant  functions  gained  constant  high 
ratings.  For  all  other  tasks  the  quality  of  machine  assistance 
improved  over  the  succession  of  the  experiments. 

The  following  Figure  10  puts  the  focus  upon  the  evaluation  of 
CAMA's  advice  and  warning  philosophy.  Advices  and  warnings 
are  issued  whenever  the  flight-deck  crew  acts  erroneously  with 
respect  to  a  CAMA-internal  model  of  what  the  pilot  is  supposed 
to  do.  Most  effort  in  modelling  pilot's  behaviour  was  made  in 
the  field  of  primary  and  secondary  flight  control  adjustments. 
Therefore,  the  evaluation  of  the  advice  and  warning  philosophy 
gives  a  good  indication  of  the  performance  of  the  pilot's 
behaviour  models  and  the  intent  and  error  recognition. 


restrained 
pleasant 
appropriate 

1  2  3  4  5  6  7 


obstrusive 

unpleasant 

inappropriate 


Figure  12:  Acceptance  of  CAMA  behaviour 

The  overall  somewhat  hesitant  acceptance  shown  in  the  results 
of  the  evaluation  (Figure  12)  may  indicate  a  certain  amount  of 
disapproval  of  new  technologies  on  behalf  of  the  pilots.  Of 
course,  a  cockpit  system  which  points  out  faulty  crew  actions 
might  be  found  sometimes  obstrusive  or  somewhat  unpleasant. 
It  might  also  be  the  outcome  of  immaturity  of  the  implemen¬ 
tation  of  certain  functions  or  procedure  models.  On  the  other 
hand  the  actions  of  CAMA  were  not  only  understood,  but 
regarded  as  being  appropriate. 

The  second  experimental  campaign  was  concluded  by  a  more 
general  subjective  evaluation.  The  pilots  were  asked  to  give 
their  opinion  on  CAMA  in  the  context  of  some  general  terms. 
Figure  13  shows  the  results.  1  indicates  a  strong  agreement  to 
the  statement,  7  means  the  strongest  possible  experession  of 
disagreement. 
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(a)  CAMA  increases  Flight  Safety 


^  I  '  " 

■  '  i  1 

'-i 

□ 

,  - 

1  1 

) 

3  4  7 

(b)  CAMA  increases  Mission  Efficiency 
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(c)  CAMA  increases  Survival  Probability 
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Figure  13:  General  Evaluation  of  CAMA 

The  results  in  Figure  13  show  that  the  potential  users  strongly 
beleive  in  the  benefit  and  an  increase  of  overall  mission  per¬ 
formance  and  success  created  by  cockpit  assistant  system  for 
military  aircraft  as  proposed  with  CAMA. 

Generally,  the  results  of  the  evaluation  of  the  questionnaires  and 
also  of  informal  statements  from  the  pilots  can  be  regarded  as 
very  good.  One  experimental  subject  stated  that  CAMA  gives 
him  the  opportunity  to  fly  even  more  perfectly  than  he  does 
anyway! 

5.2  Objective  evaluation  statements 
In  this  study  objective  evaluations  were  done  only  on  the  level 
of  the  functional  modules  (e.g.  concerning  the  Pilot  Behaviour 
Model  and  the  Pilot  Intent  and  Error  Recognition).  A  detailed 
description  of  respective  results  would  be  beyond  the  scope  of 
this  paper.  (See  [14]  and  [15]  for  details.) 


However,  the  following  Table  3  gives  an  idea  of  a  comparison 
of  typical  reaction  times  (RT)  between  CAMA  and  an  unas¬ 
sisted  crew  (estimated). 


Situation 

Appropriate 

Reaction 

RT[s] 

unassisted 

with  CAMA 

new  threat 
pops  up 

check  current 
flight  plan 

12 

2 

threat  affects 
current  plan 

check  re-planning 
required 

5 

<  1 

destination 

changed 

IFR  re-planning 

>30 

2 

tactcal  transit 
corridor  chng. 

optimal  trajectory 
re-planning 

N.A. 
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Table  3:  Comparison  between  reaction  times 


It  is  obvious  that  CAMA  takes  a  lot  of  mental  workload  from 
the  cockpit  crew  by  applying  its  highly  automated  functions  or 
simply  enables  the  crew  to  take  advantage  out  of  its  complex 
computations  in  the  context  of  situation  interpretation  and 
conflict  resolution  tasks. 

Furthermore,  continuous  observations  of  the  experimental  task 
performance  led  to  some  optimisation  steps  between  the  two 
evaluation  periods: 

•  The  presentation  of  flightplan  proposal  on  the  navigation 
display  was  found  to  be  irritating  for  the  pilot  in  certain 
situations.  Therefore  the  depiction  was  moved  to  the  sec¬ 
ondary  display. 

•  A  pure  visualisation  of  terrain  elevation  on  the  primary 
flight  display  was  found  to  be  insufficent  for  terrain  relief 
perception  and  low  level  flight  guidance.  As  a  result  three- 
dimensional,  triangulated  feature  data  and  textures  were 
added. 


6  CONCLUSIONS 

The  results  of  the  experimental  evaluations  of  the  Crew  As¬ 
sistant  Military  Aircraft  yield  a  wide  variety  of  approaches  and 
solutions  in  the  field  of  crew  assistance  for  tactical  flight  mis¬ 
sions.  The  next  two  subsections  conclude  the  paper  by  giving  an 
overview  and  evaluation  of  the  present  work  reported  in  this 
paper,  and  pointing  out  some  future  prospects  for  forthcoming 
developments. 

6.1  Present  work 

In  the  present  work  the  functions  required  in  order  to  construct 
an  assistant  system  for  military  aircraft  operations  were  derived 
from  the  tasks  relating  to  tactical  mission  management.  The 
result  of  this  analysis  shows  that  functions  for  situation  inter¬ 
pretation  /  assessment,  conflict  detection  /  resolution  and  a  man- 
machine  interaction  management  make  up  the  core  of  such  an 
assistant  system.  Taking  the  requirements  of  human-centered 
automation  into  consideration,  the  cognitive  assistant  system 
CAMA  provides  well  founded  solutions.  The  integration  of 
tactical  mission  management  functions  into  CAMA  completes 
the  system,  so  that  an  autonomous  recognition  of  crew  intent 
and  errors  as  well  as  the  detection  of  flight  plan  conflicts  can 
trigger  dedicated  mechanisms  of  conflict  resolution. 

CAMA  and  its  various  assistant  functions  were  throroughly 
evaluated  during  two  experimental  campaigns  utilising  a  flight 
simulator.  A  total  of  ten  professional  military  pilots  performed 
complex  missions  in  the  simulator,  assisted  by  the  Crew  As¬ 
sistant  Military  Aircraft.  In  the  de-briefmg  sessions  the  pilots 
gave  extremely  positive  ratings  on  the  system,  its  contribution 
to  situation  awareness,  the  quality  of  the  assistant  functions,  and 
the  degree  of  acceptance  of  such  an  electronic  crew  member. 
The  effect  of  the  minimum  amount  of  training  on  the  system,  in 
particular  on  the  easyness  of  handling,  is  remarkable. 

6.2  Future  prospects 

After  the  successful  experimental  simulator  evaluation,  a  flight 
trial  campaign  with  CAMA  is  under  contract  and  scheduled  for 
early  2000.  Further  developments  will  focus  upon  the 
integration  of  imaging  sensor  information  into  the  three- 
dimensional  flight  guidance  display  in  order  to  achieve  an 
enhanced  vision  system. 
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1.  INTRODUCTION 

Battlefield  helicopters  will  form  an  important  element  of  the 
Digitized  Battlespace.  Helicopter  aircrew  will  be  presented 
with  large  amounts  of  data  from  both  on-board  sensors  and 
other  units.  The  way  in  which  this  information  is  presented 
to  the  crew  will  have  a  critical  impact  on  operational 
effectiveness.  This  paper  describes  the  a  trial  carried  out  by 
the  Defence  Evaluation  and  Research  Agency  (DERA)  for 
UK  MoD,  to  assess  the  impact  of  providing  Tactical 
Situation  Displays  (TSD)  of  varying  degrees  of  data  fusion 
and  complexity. 

Previous  trials  [1,  2]  assessed  the  impact  on  operational 
effectiveness  of  the  provision  of  tactical  information  in  the 
helicopters.  These  concluded  that  the  provision  of  tactical 
information  improved  operational  effectiveness,  survivability 
and  situational  awareness.  Instances  of  fratricide  reduced 
dramatically.  These  parameters  improved  still  further  when 
on-board  sensor  data  was  fused  with  the  tactical  display. 

The  two  previous  trials  were  undertaken  prior  to  the  selection 
of  the  WAH-64  as  the  UK  Army’s  Attack  Helicopter.  It  was 
considered  important  to  re-assess  the  conclusions  of  the 
earlier  trials  in  the  light  of  the  capabilities  of  the  WAH-64. 
This  trial  was  designed  to  assess  the  effect  on  operational 
effectiveness  of  providing  on-board  tactical  information 
displays  of  varying  degrees  of  integration  to  the  Commander 
of  the  WAH-64.  A  fourth  trial,  due  to  take  place  in  1998, 
will  examine  the  impact  of  different  levels  and  accuracies  of 
tactical  information. 

2.  HELICOPTER  SIMULATOR  TRIAL 

2.1  Hovers 

The  trial  was  conducted  using  the  HOVERS  real-time  man- 
in-the-loop  helicopter  mission  simulator.  This  is  a  fixed  base 
workstation  based  facility.  HOVERS  models  helicopter 
systems  (sensors,  weapons,  flight  dynamics),  a  synthetic 
battlespace  environment,  and  semi-automated  Computer 
Generated  Forces  (CGF). 

Cockpits,  with  stations  for  the  Pilot  and  the  Commander,  are 
located  in  separate  rooms.  Each  simulated  helicopter  is 
flown  using  conventional  cyclic  stick,  collective  lever  and 
yaw  pedals.  The  Pilot’s  view  is  provided  by  computer¬ 
generated  terrain  images  displayed  on  three  large  monitors. 
This  view  is  also  visible  to  the  Commander.  Both  Pilot  and 
Commander  also  have  head-down  displays  showing  basic 
flight  and  mission  related  information.  Controls  include 
touchscreen  buttons,  switches  and  joysticks.  The  Pilot  and 
Commander  can  communicate  with  each  other,  with  other 
helicopter  crews  and  with  the  tactical  controller. 


2.2  Cockpit  configuration 

The  models  developed  for  this  trial  were  generic 
representations  of  the  systems  which  will  be  fitted  to  the 
WAH-64  at  its  In  Service  Date  (ISD),  together  with 
enhancements  and  additional  systems  likely  to  fitted  by 
2010.  The  2010  timeframe  was  selected  as  the  earliest  date 
at  which  a  highly  integrated  TSD  could  be  fitted  to  the 
WAH-64. 

It  was  assumed  that  tactical  information  from  the  digitized 
battlespace  would  be  displayed  on  the  TSD,  overlaid  on  a 
digitized  map,  together  with  information  from  on-board 
sensors.  The  TSD  configurations  are  described  in  Section 
2.3. 

The  three  sensor  systems  modelled  were  a  Fire  Control  Radar 
(FCR)  (used  for  target  classification  and  prioritisation),  a 
Radio  Frequency  Interferometer,  and  a  Target  Acquisition 
Designation  System  (TADS).  The  TADS  model  consisted  of 
an  Infra-Red  (IR)  sensor,  Laser  Range  Finder  and  Image 
Autotracker. 

The  helicopter  was  provided  with  three  weapons  systems. 
Generic  models  of  Radio  Frequency  (RF)  missiles.  Cannon 
and  IR  Fire  and  Forget  Air  to  Air  Missiles  were 
implemented. 

A  Combat  Identification  model  was  integrated  with  the 
TADS  and  FCR  to  prevent  RF  or  IR  missiles  from  being 
targeted  at  friendly  vehicles. 

A  generic  integrated  DAS  suite  was  modelled,  consisting  of  a 
Radar  Warning  Receiver,  Laser  Warning  Receiver,  Omni¬ 
directional  RF  Jammer,  Missile  Warning  System,  Directional 
IR  jammer  and  Flares. 

2.3  Tactical  Situation  Display  Configurations 

The  trial  used  three  different  configurations  of  the 
Commander’s  TSD,  while  the  Pilot’s  display  remained 
unchanged.  The  level  of  fusion  of  on-board  sensor  data  and 
imported  tactical  information  on  a  digitized  map  differed  in 
each  configuration.  Some  additional  display  functionality 
was  also  added.  The  three  configurations  were: 

•  Baseline:  On-board  sensor  data  as  available  at  ISD 
overlaid  on  a  digitized  map  displaying  tactical 
information  from  the  digitized  battlespace; 

•  Improved:  Some  integration  of  on-board  sensor  data 
with  the  tactical  information  from  the  digitized 
battlespace; 

•  Fused:  Complete  fusion  of  on-board  sensor  data  and 
imported  tactical  information  to  provide  a  simpler,  more 
accurate  TSD. 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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2.3. 1  Baseline  display 

The  baseline  display  comprised  the  following: 

•  Digitized  map:  With  tactical  symbology  overlays; 

•  Sensor  data:  Similar  to  that  expected  on  the  TSD  on  the 
WAH-64.  Targets,  allegiance,  ownship  position,  DAS 
warnings,  and  FCR  display; 

•  Tactical  information  updates:  Accurate,  up  to  date 
information  on  Blue  Forces.  Incomplete  information  on 
Red  Forces,  subject  to  errors  in  position,  vehicle  type 
and  staleness.  Stalenness  portrayed  by  fading  vehicle 
symbols; 

•  Controls:  Similar  to  those  expected  on  the  WAH-64  at 
ISD.  The  map  could  be  manipulated  to  alter  scale, 
intensity  under  the  tactical  symbology  overlay,  and 
centre  position  (on  ownship  symbol  or  grid  reference). 

2.3.2  Improved  display 

The  improved  display  provided  the  Commander  with  all  of 
the  information  and  options  that  were  available  in  the 
baseline,  together  with  the  following  additional  features: 

•  Limited  integration  of  on-board  sensor  data  with  tactical 
data; 

•  Intervisibility  calculations; 

•  Additional  map  manipulation  functionality. 

Limited  integration  provided  between  the  on-board  sensor 
data  and  the  tactical  data  provided  the  following 
improvements  in  tactical  symbology; 

•  Listing  of  the  last  position  measured  with  the  laser 
range  finder; 

•  Lines  on  the  map  to  indicate  the  range  and  bearing  of 
the  laser  range  finder; 

•  Display  of  Laser  Warner  Receiver  warnings  on  the 
border  of  the  map  together  with  other  DAS  warnings, 
rather  than  on  the  map  itself; 

•  Lines  from  the  ownship  symbol  to  DAS  symbols  on  the 
edge  of  the  TSD; 

•  Colour  coded  DAS  warnings  to  indicate  type; 

•  TADS  and  FCR  sensor  footprints,  including 
intervisibility. 

Intervisibility  information  was  calculated  in  real  time  using 
terrain  heights  and  cultural  features.  The  following  could  be 
displayed  on  the  map: 

•  Line  of  sight  areas  to  a  fixed  maximum  range; 

•  Sensor  foot  prints  showing  angular  coverage  of  the 
sensor  and  the  area  of  ground  to  which  the  sensor  had 
line  of  sight; 

•  Virtual  helicopter  positions:  By  selecting  a  point  on  the 
map  and  a  height,  the  Commander  could  display  the 
area  of  ground  that  would  be  visible  to  the  helicopter  if 
it  moved  to  that  position; 

•  Virtual  ground  force  positions:  By  selecting  a  point  on 
the  ground,  the  Commander  could  display  the  area  over 
which  a  ground  platform  would  have  line  of  sight  to 
helicopters  at  various  heights. 

Additional  map  manipulation  functionality  was  provided  as 
follows: 


•  Map  scrolling; 

•  Additional  scale  options; 

•  Additional  orientation  options:  Aircraft  referenced. 
North  up,  FCR  up  and  TADS  up. 

2.3.3  Fused  display 

The  fused  display  had  the  following  key  features: 

•  Complete  fusion  of  tactical  data  from  the  digitized 
battlespace  with  on-board  sensor  information.  This 
simplified  the  picture.  The  accuracy  of  the  information 
was  also  improved,  as  the  single  most  accurate  and  up  to 
date  source  of  vehicle  position  information  was  used  to 
update  the  display; 

•  DAS  warnings  associated  with  specific  platforms  rather 
than  bearings; 

•  Intervisibility  calculations  and  map  functionality  as  for 
the  improved  display. 

2. 3. 4  Pilot 's  display 

The  Pilot’s  display,  which  remained  unchanged,  comprised 
the  following: 

•  North  up  single  scale  digitized  map; 

•  Baseline  TSD; 

•  Tactical  information  updates  as  provided  for  the 
Commanders. 

2.4  Scenarios  and  threat  environment 

Army  Air  Corps  (AAC)  personnel  designed  eight  different 
scenarios.  The  scenarios  were  designed  to  provide  a  high 
intensity  threat  environment.  In  each  scenario  a  pair  of 
manned  (Blue)  HOVERS  helicopters  was  tasked  with 
seeking  and  destroying  enemy  (Red)  CGF.  One  helicopter 
assumed  the  role  of  patrol  commander.  In  the  event  of  one 
of  the  helicopters  being  killed,  the  second  was  to  proceed 
with  the  mission  alone. 

Missions  were  conducted  with  sensor  and  weapon  systems 
appropriate  to  the  2010  timeframe.  The  ground  force  threat 
was  formed  into  groups  comprising  main  battle  tanks,  air 
defence  units  (ADUs)  and  armoured  personnel  carriers 
carrying  man  portable  air  defence  systems.  Generic  CGF 
attack  helicopters  provided  an  air  threat. 

2.5  Trial  procedure 

AAC  personnel  flew  72  missions  over  a  3  week  period.  The 
HOVERS  military  co-ordinator  acted  as  the  tactical 
controller.  An  initial  period  of  structured  training  was 
provided  for  the  trials  aircrew  to  ensure  familiarity  with  the 
aircraft  systems.  Crews  were  issued  with  orders  prior  to  each 
mission,  and  allocated  time  to  plan  the  mission. 

Data  were  logged  during  each  mission  for  later  analysis.  The 
key  measures  of  effectiveness  for  these  attack  missions  were 
the  survivability  of  Blue  helicopters  and  the  number  of 
enemy  vehicles  destroyed.  Crews  also  completed  debriefing 
questionnaires  to  provide  a  subjective  analysis  of  workload 
[3]  and  situational  awareness  [4]. 
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3.  RESULTS 


3. 1. 1  Reduction  in  kills  byADUs 


3.1  Survival  probability 

Figure  3.1  shows  that  survival  probability  of  the  Blue 
helicopters  increased  slightly  for  both  the  improved  and 
fused  configurations. 
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Figure  3.1  Blue  helicopter  survival  probability 

As  shown  in  Figures  3.2  and  3.3,  a  significant  reduction  in 
the  number  of  kills  by  ADUs  was  balanced  by  an  increase  in 
the  number  of  kills  by  Red  helicopters. 
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Figure  3.2  Probability  of  Blue  helicopter  being  killed  by 
ADUs 


Probability  of  the  blue  helicopter 
being  killed  by  a  red  helicopter 


The  improvements  in  survival  probability  against  ADUs 
were  analysed  further.  Figure  3.4  shows  that  there  was  a 
reduction  in  the  number  of  sightings  of  Blue  helicopters  by 
Red  ADUs,  leading  to  a  drop  in  the  number  of  missiles  fired 
at  Blue  helicopters.  Intervisibility  information  and  increased 
confidence  in  the  accuracy  of  the  more  integrated  displays 
allowed  Commanders  to  plan  more  covert  routes  to  firing 
positions. 
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Figure  3.4  Number  of  sightings  of  Blue  helicopters  by 
ADUs 


Figure  3.5  shows  that  the  effectiveness  of  ADU  missiles  fell 
with  the  more  integrated  displays.  Crews  used  the 
intervisibility  options  to  improve  their  use  of  terrain, 
selecting  firing  positions  from  which  it  would  be  easier  to 
break  missile  lock.  Increasing  integration  of  the  displays 
improved  the  accuracy  of  the  intervisibility  options.  The 
increasing  integration  of  DAS  warnings  with  the  map  further 
increased  crew  awareness  of  the  threat. 


Effectiveness  of  ADU  missiles 
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Figure  3.5  Effectiveness  of  ADU  missiles 


3. 1.2  Increase  in  kills  by  Red  helicopters 

The  increase  in  the  number  of  kills  by  Red  helicopters  was 
demonstrated  by  a  fourfold  increase  in  the  probability  that  a 
missile  fired  at  the  Blue  helicopter  by  the  enemy  helicopters 
would  successfully  destroy  it.  This  is  shown  in  figure  3.6. 


Figure  3.3  Probability  of  a  Blue  helicopter  being  killed 
by  a  Red  helicopter 


17-4 


Effectiveness  of  red  helicopter 
missiles 


Figure  3.6  Effectiveness  of  Red  helicopter  missiles 


Observation  of  the  crews  and  subsequent  analysis  of  the 
video  tapes  recorded  during  the  trial  showed  that  this  was 
due  to  a  combination  of  factors: 


•  Commanders  tended  to  focus  their  attention  on  the  map 
display,  which  did  not  show  the  position  of  fast  moving 
enemy  helicopters  with  great  accuracy.  Commanders 
focused  on  the  ground  threats  at  the  expense  of 
searching  for  air  threats; 

•  Commanders  tended  to  rely  on  the  intervisibility 
calculations  to  determine  when  it  was  worth  scanning 
for  ground  vehicles  with  the  FCR.  Less  frequent  use  of 
the  FCR  with  the  improved  and  fused  configurations 
reduced  long  range  warning  of  an  enemy  helicopter 
approach; 

•  Commanders  selected  their  observation  points  using  the 
virtual  helicopter  and  virtual  enemy  position  functions. 
Crews  were  then  confident  that  they  had  selected  a 
suitable  and  covert  position  from  which  to  observe 
enemy  ground  vehicles.  With  increased  confidence  in 
the  safety  of  the  observation  point,  they  appeared  to  be 
less  vigilant  in  detecting  air  threats,  despite  the 
availability  of  DAS  warnings. 


3.2  Destruction  of  enemy  vehicles 


Despite  the  increase  in  covertness,  no  significant  change  was 
observed  in  the  total  number  of  enemy  vehicles  killed.  This 
indicated  that  improvements  in  survivability  were  achieved 
without  impacting  operational  effectiveness.  Moreover,  the 
proportion  of  ADUs  (priority  targets)  destroyed  rose  sharply 
with  increasing  integration,  as  did  the  ability  of  Blue 
helicopters  to  fire  first. 


Probability  that  the  first  ground 
vehicle  killed  was  an  ADU 
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Figure  3.7  Probability  that  the  first  ground  vehicle  killed 
was  an  ADU 


There  were  two  primary  reasons  for  the  lack  of  variation  in 
the  total  number  of  vehicles  destroyed. 

•  Once  the  Blue  helicopter  reached  the  point  of  firing,  the 
FCR  was  used  to  prioritise  the  targets.  The  number  of 
vehicles  destroyed  was  largely  determined  by  the 
performance  of  the  FCR  and  the  missiles  rather  than  by 
use  of  the  TSD. 

•  The  Blue  helicopters  did  not  have  significantly  longer  to 
destroy  the  enemy  vehicles  as  the  survival  probability 
and  average  mission  duration  were  only  slightly 
increased  in  the  improved  and  fused  configurations. 

3.3  Other  factors 

The  length  of  time  after  take  off  before  crew  fired  their  first 
shot  was  significantly  increased  for  the  fused  configuration 
over  the  baseline  and  improved  configurations.  This 
appeared  to  be  due  to  an  increase  in  planning  time  in  the 
field  as  tactical  information  was  updated  by  more  accurate 
information  from  on-board  sensors. 

However,  the  time  required  for  mission  planning  prior  to  the 
mission  increased  for  both  the  improved  and  fused 
eonfigurations. 

Commanders  experienced  considerable  frustration  when  false 
alarms  and  errors  appeared  in  the  improved  and  fused 
displays.  It  appeared  that  their  expectations  of  the  accuracy 
of  the  system  were  too  high.  A  guide  to  the  likely  magnitude 
of  the  error  in  position  would  have  been  useful. 

Pilots  did  not  feel  able  to  contribute  fully  to  the  mission. 
Effective  teamworking  within  the  cockpit  was  hampered  due 
to  the  disparity  in  the  information  displayed  to  the  two  crew 
members. 

4.  CONCLUSIONS 


4.1  The  following  conclusions  were  drawn; 

•  Increasing  the  level  of  fusion  of  on-board  sensor  data 
and  imported  tactical  information  to  provide  a  simpler 
tactical  situation  display  improved  crew  planning  and 
survivability; 

•  Comprehensive  training  in  the  operational  use  of 
advanced  fused  tactical  situation  displays  will  be 
essential  if  their  full  benefit  is  to  be  realised; 

•  Similar  information,  tailored  to  roles,  should  be 
provided  to  both  crew  members  so  that  Pilot  and 
Commander  can  communicate  effectively  and  take  on 
an  appropriate  share  of  the  workload; 

•  On-board  planning  tools  and  additional  display 
functionality,  particularly  intervisibility  options,  can 
augment  the  benefits  of  the  fused  display. 

5.  NOMENCLATURE 

AAC  UK  Army  Air  Corps 

ADU  Air  Defence  Unit 

CGF  Computer  Generated  Forces 

DAS  Defensive  Aids  Suite 

FCR  Fire  control  radar 

ISD  In-Service  Date 

TADS  Target  Acquisition  Designation  System 


TSD  Tactical  Situation  Display 

WAH-64  Westlands  Apache  Attack  Helicopter 
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ABSTRACT  LIST  OF  ABBREVIATIONS 


In  the  NATO  E-3A  Mid-Term  Modernisation  Programme, 
the  Airborne  Systems  Division  (ASD)  of  Daimler-Benz 
Aerospace  was  selected  in  a  competitive  process  to 
contribute  three  work  packages.  These  packages  comprise 
the  Multi-Sensor  Integration,  Tracking  and  Identification 
Software  as  well  as  the  AWACS  Mission  Computer  and 
the  Multi-Sensor  Integration  Computer  Hardware.  When 
the  selection  process  for  the  individual  work  packages 
was  completed,  the  Statement  of  Work  (SOW)  for  DAS  A 
ASD  was  extended  in  order  to  deliver  a  single  integrated 
subsystem  comprising  both,  hardware  and  software. 

The  requirements  in  the  NATO  E-3A  Mid-Term 
Modernisation  Programme  call  for  real-time  integration  of 
all  available  information  pertaining  to  one  real  world 
object,  in  particular  geometric,  kinematics  and  signature 
data.  The  new  system  shall  improve  tracking  quality  and 
additionally  give  automatic  identification/classification  of 
targets  in  order  to  significantly  reduce  operator  workload. 

The  Multi-Sensor  Integration  function  as  it  will  be 
realized  for  AWACS  will  use  data  of  the  primary 
surveillance  radar,  secondary  surveillance  radar  (IFF), 
data  of  passive  electronic  support  measures  and  crosstold 
data  of  several  links  including  Link- 16.  All  these  data  are 
input  to  a  data  fusion  process  in  order  to  generate  a  clear 
and  tme  picture  of  the  real-world  situation. 

This  paper  will  present  an  overview  of  the  technical 
concepts  and  the  selected  solutions  for  the  multi-sensor 
integration  task.  It  will  show  how  the  multi-sensor 
integration  is  performed  and  what  the  critical  issues  are  in 
the  course  of  data  fusion.  Furthermore  it  will  briefly 
outline  the  test  concepts  which  will  be  used  for  the  final 
acceptance  of  the  system. 


AMC  AWACS  Mission  Computer 
AMCP  AWACS  Mission  Computer  Program 
ASD  Airborne  Systems  Division  (a  division  of 
Daimler-Benz  Aerospace) 

CORBA  Common  Object  Request  Broker  Architecture 

COTS  Commercial-of-the-shelf 

CSCI  Computer  Software  Configuration  Item 

DASA  Daimler-Benz  Aerospace 

DSI  Distributed  Software  Infrastructure 

EMD  Engineering  and  Manufacturing  Development 
ESC  Electronic  Systems  Center 

ESM  Electronic  Support  Measures 

GPS  Global  Positioning  System 

IDBO  Identification  by  Origin 
IPT  Integrated  Product  Team 

LAN  Local  Area  Network 

MSI  Multi-Sensor  Integration 

MSIC  Multi-Sensor  Integration  Computer 

MSICP  Multi-Sensor  Integration  Computer  Program 

NIC  Network  Interface  Card 

RMA  Removable  Memory  Assembly 

SCSI  Small  Computer  System  Interface 
STRP  Supplier  Technical  Requirements  Package 

TADS  Test  and  Development  Software 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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1  fflSTORICAL  BACKGROUND 

The  definition  of  the  NATO  E-3A  Mid-Term 
Modernisation  Programme  began  in  the  first  years  of  the 
nineties  when  the  military  community  was  confronted 
with  a  radically  changed  political  situation.  This  new 
situation  resulted  in  changed  tasks  for  all  forces  of  the 
NATO  countries.  Especially,  the  former  relatively  stable 
situation  with  well  known  border  lines  between  the 
eastern  and  western  block  was  not  in  place  any  more. 
Consequently,  the  surveillance  task  has  become  much 
more  demanding  and  challenging.  Airborne  surveillance 
systems  now  have  to  cope  with  a  much  higher  number  of 
inputs  stemming  from  ground,  maritime  or  airborne 
targets  which  are  moving  in  the  area  of  many  different 
smaller  countries.  This  situation  imposes  special  high 
challenges  to  the  identification  task  because  during  peace 
keeping  missions  for  example  the  situation  may  vary  often 
and  quickly. 

Therefore,  the  planning  for  the  Mid-Term  Programme 
asks  for  much  higher  flexibility  and  adaptability  to  the 
actual  situation  in  comparison  to  today’s  systems.  In 
parallel,  the  operator  workload  shall  be  reduced  from 
today’s  levels,  even  in  much  denser  and  much  more 
complex  scenarios. 


2  PROGRAMME  OVERVIEW  AND  USER 
EXPECTATIONS 

The  Mid-Term  Modernisation  Programme  will  enhance 
the  capabilities  of  NATO  E-3A  aircraft  in  a  number  of 
major  system-related  areas.  New  equipment  will  be 
installed  for: 

•  Digital  communication  including  satellite 
communication  systems  with  automatic  record  and 
replay  of  all  data 

•  Broad  spectrum  VHP  radios 

•  GPS-based  navigation  to  provide  accurate  passive 
own-ship  positioning 

•  Enhanced  identification  capabilities  to  cope  with  new 
international  air  traffic  regulations 

•  Real-time  integration  of  all  available  sensor  data  to 
provide  timely  and  accurate  situation  displays 

•  Operator  workstations  containing  state-of-the-art  flat 
panel  situation  displays 

•  New  computers  based  on  COTS  products  employing 
an  open  systems  architecture,  also  to  provide  a  basis 
for  future  systems  upgrades. 

These  enhancements  will  significantly  contribute  to 
enable  AW  ACS  handling  the  complex  scenarios  in  the 
future  and  to  further  reduce  operator  workload. 


The  users  demand  a  system  which  can  be  adapted  to 
specific  requirements  driven  by  actual  tasks  and  scenarios. 
This  requirement  lead  to  the  concept  of  a  generic  software 
package  which  can  be  „prograinmed“  by  a  set  of  mission 
preparation  data  to  be  loaded  into  the  system  prior  to 
operation.  Also,  the  system  can  handle  several 
identification  schemes  in  parallel. 


3  THE  MULTTS-SENSOR  INTEGRATION  SYSTEM 

3.1  MSI  System  Overview 

At  the  end,  the  ultimate  goal  of  all  multi-sensor 
integration  measures  is  to  generate  a  precise  picture  of  the 
according  real  world  scenario.  The  system  has  to  merge 
all  available  inputs  that  pertain  to  one  target  to  only  one 
object  in  the  picture  on  the  display.  This  picture  must 
contain  only  real  and  no  false  targets,  all  correctly 
identified  at  the  correct  locations.  This  goal  can  only  be 
achieved  by  a  careful  harmonization  of  the  sensor 
integration  software  with  the  high  performance  of  the 
contributing  sensors.  The  term  Multi-Sensor  Integration 
comprises  essentially  two  distinct  functions,  which  are 
Multi-Sensor  Tracking  including  the  correlation  function 
and  the  Multi-Sensor  Identification. 

The  CSCI MSICP  will  provide  multi  sensor  tracking  and 
identification  functionality  within  the  NATO  Mid-Term 
Modernisation  Programme  of  the  NATO  E-3A  aircraft. 
The  MSICP  will  consist  of  the  following  parts: 

•  MSI  Tracking 

•  MSI  Identification 

•  MSI  Manager 

The  MSI  Tracking  and  Identification  functions  provide 
the  basic  operational  functionality  of  the  MSICP.  The 
MSI  Manager  includes  functionality  mainly  in  the  area  of 
control  and  monitoring,  communication,  redundancy 
support  and  test  and  maintenance  provisions  for  the 
MSICP. 

The  MSICP  software  will  be  coded  in  ADA95  and 
C/C-I-I-.  It  will  be  implemented  according  to  POSIX 
standards  and  executable  on  a  SUN/SPARC  compatible 
computer  miming  a  Solaris  2.x  operating  system. 

The  improved  MSI  function  will  be  hosted  on  a  redundant 
state-of-the-art  hardware  computer  architecture,  the 
MSIC.  It  will  be  integrated  into  the  overall  system  by  a 
direct  interface  to  the  AMCP,  which  is  not  part  of  the 
work  share  of  DASA  Airborne  Systems.  This  AMCP  will 
again  be  hosted  on  a  redundant  computer  system  of  a 
similar  architecture,  the  AMC. 
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Figure  2-1 

3.2  The  Computer  System 

In  the  initial  STRP  a  proposal  for  the  MSI  computer  and 
an  adjunct  computer  to  the  existing  CC-2A  computer  was 
requested.  During  the  following  concept  refinement,  it 
was  decided  to  remove  the  current  CC-2A  computer 
completely  and  replace  it  by  an  all  new  AWACS  mission 
computer  in  order  to  unify  systems  architecture  and  to 
minimize  life-cycle  cost. 

Both,  the  AWACS  Mission  Computer  and  the  Multi- 
Sensor  Integration  Computer  are  fully  redundant 
computer  systems  with  cross  monitoring  between  the 
active  and  passive  computer.  In  case  of  a  failure  in  the 
active  computer,  the  passive  computer  will  be  able  to  take 
over  immediately,  because  of  permanent  input  of  sensor 
data  and  checkpointing  of  results  into  the  passive 
computer.  This  approach  gives  extremely  high  system 
availability,  i.e.  under  normal  operational  conditions  the 
air/surface/ground  picture  never  will  be  lost. 

The  AMC  and  MSIC  are  computers  based  on  COTS 
elements.  The  new  computer  systems  use  Ultra-Sparc  2 
processor  chips  from  SUN  computers  of  the  latest 
generation  and  they  are  build  as  industry  standard  VME- 
Bus  cards  and  are  software  compatible  with  commercial 
SUN  workstations.  The  industry  standard  VME-Bus 
architecture  allows  quick  and  cost  effective  integration  of 
external  interfaces  such  as  Mil-Bus  I/F  and  special  build 
interfaces.  All  computers  contain  a  system  surveillance 
board  with  temperature  sensors  and  voltage  monitoring. 

For  the  purpose  of  data  exchange,  the  computers  will  be 
linked  via  a  Gigabit  Ethernet  designed  to  be  fully 
redundant.  Again  standard  COTS  communication 
protocols  will  be  used  for  all  communication  via  this 
network. 

The  AMC  is  a  single  processor  computer  equipped  with 
256  Mbytes  of  Memory.  The  AMC  computers  have  access 
via  two  SCSI  busses  to  the  RMA  hard  disks  and  CD- 
ROM  changers.  In  case  of  failure,  the  active  computer  is 
discormected  from  the  SCSI  busses  and  the  passive  one 


takes  over  the  disks  (see  figure  3.2-1  AMC).  The  MSIC  is 
a  dual  processor  computer  with  again  256  Mbytes  of 
memory.  It  has  no  access  to  external  devices  (see  figure 
3.2-2  MSIC). 

Both  computer  systems  (AMC  and  MSIC)  have  access  to 
a  private  local  area  network  (LAN)  to  perform 
communications  related  to  health  monitoring.  This  LAN 
is  used  exclusively  for  that  purpose. 

All  computer  hardware  is  housed  in  industry  standard  19- 
inch  racks.  Each  computer  is  housed  in  its  own  rack;  all 
hard  disks  and  CD-ROM  changers  are  housed  in  one  rack. 
Hence,  the  total  number  is  five  racks. 

The  application  software  development  takes  place  on 
regular  workstations  with  the  standard  UNIX  based 
operating  system  Solaris  and  a  state-of-the-art  software 
engineering  environment. 

The  CSCI AMC/MSIC  System  Software  delivered  by 
ASD  will  provide  all  operating  system  functionality 
needed  to  operate  the  AMC  and  MSIC  on  a  Sun/Solaris 
2.x  operating  system  basis.  This  CSCI  will  consist  of 
COTS  software  products  configured  for  use  within  the 
DASA  delivered  computer  hardware  components. 

On  top  of  this  operating  system  level  software  a 
Lockheed-Martin  provided  Distributed  Software 
Infrastracture  (DSI)  middleware  will  be  responsible  for 
communication  issues  within  this  system.  The  DSI  will  be 
CORBA  2.0  compliant. 


3.3  The  Multi-Sensor  Tracking  System 

The  Multi-Sensor  Tracking  is  capable  to  use  sensor  inputs 
from  a  wide  variety  of  different  sensors  which  are  the 
primary  AWACS  radar,  the  SSR/IFF  sensor,  the  ESM 
system  and  crosstold  sensor  inputs  via  the  various  tactical 
data  links.  It  is  based  upon  Multi-Model  technology  which 
guarantees  in  the  various  maneuver  conditions  optimal 
track  stability  and  continuity.  New  tracks  will  be  initiated 
automatically  either  based  on  data  inputs  from  active 
radars  or  from  passive  sensors  only.  The  tracking  process 
is  capable  to  perform  self-triangulation  based  on  passive 
strobes  from  onboard  sensors.  Additionally,  ESM  reports 
from  tactical  data  links  are  used  together  with  reports 
from  onboard  sensors  to  perform  multi-sensor  cooperative 
passive  tracking.  Deghosting  is  done  automatically. 

The  functions  of  the  Tracking  system  are  grouped  in  the 
following  categories  (see  Figure  3.3-1); 

•  Data  preprocessing 

•  Correlation/Association 

•  Track  Update 

•  Track  List  Management 
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•  Post  processing 

The  preprocessing  comprises  functions  like  ordering  of 
measurement  data  in  a  timeline  related  sequence, 
performing  coordinate  system  transformation,  etc..  In  this 
processing  bias  compensation  is  performed  for  all  input 
sensors. 

In  the  correlation  function,  either  the  classical  nearest 
neighbor  algorithm  or  the  advanced  Multi-hypothesis 
concepts  are  applied.  Which  concept  will  be  applied  is 
dependent  on  the  actual  complexity  of  the  scenario  in  the 
vicinity  of  the  current  measurement/track  position. 
Adaptive  selection  of  the  most  suitable  correlation 
algorithm  will  significantly  reduce  processor  load.  In 
doing  so,  maximum  track  continuity  will  be  provided 
under  all  target  conditions,  and  hence,  track  identification 
will  be  preserved  through  complicated  target  track 
crossing  situations.  Furthermore,  ESM-based  signature 
data  and  cross  correlation  between  ESM-  and  kinematic 
data  will  also  be  used  for  the  final  report/track  pairing. 

The  operator  can  at  any  time  either  prohibit  a  selected 
correlation  or  enforce  a  correlation. 

The  track  update  function  is  executed  for  each  track 
which  has  been  associated  to  a  target  report.  Depending 
on  the  type  of  the  target  report  either  a  kinematic  and/or  a 
attribute  update  is  performed  for  every  model  which  is 
currently  applicable  to  the  track.  Kinematic  updating  is 
done  by  means  of  a  Kalman  component  filter,  whose 
peculiar  feature  is  that  updating  of  the  state  vector  is 
performed  for  each  measurement  component  separately. 

A  major  advantage  of  this  Kalman  component  filter  is  that 
even  under  jammed  or  distorted  conditions  all  available 
useful  measurement  information  will  be  utilized.  Attribute 
updating  is  performed  by  keeping  record  of  attribute  data 
of  target  reports  incorporated  into  the  track  (see  Figure 
3.3-2). 

The  Track  List  Management  contains  functions  like  track 
initiation,  track  deletion,  track  prediction  and  bias  error 
estimation.  Track  initiation  is  performed  on  target  reports 
for  which  no  correlation  was  found.  They  will  be  initiated 
as  tracks  and  labeled  as  potential  track.  Initiation  of  tracks 
will  not  only  be  performed  on  sensor  data  from  active 
sensors  but  also  on  data  from  passive  sensors.  Track 
deletion  will  be  applied  to  tracks,  whose  quality  figure 
recedes  below  a  certain  threshold.  These  deletion 
thresholds  automatically  adapt  to  environment  parameters 
and  to  the  different  track  states.  Track  prediction  is 
performed  for  the  time  when  the  next  sensor  update  is 
expected.  The  predicted  position  is  input  to  the  correlation 
function  the  next  time.  The  bias  error  estimation  function 
analyzes  measurement  residuals  of  each  sensor  with 
respect  to  the  system  track.  In  order  to  maximize  overall 
track  quality  and  accuracy,  possible  offsets  between  all 
sensors  (onboard  and  offboard)  are  monitored  and 
estimated  continuously  in  background.  If  necessary, 
observed  offsets  are  compensated  automatically  by  an 
adaptive  logic. 


In  the  post  processing,  coordinate  system  transformations 
are  performed  to  adapt  the  tracker  to  the  external  system. 
Finally,  data  is  converted  to  external  data  formats. 

In  summary,  the  functions  performed  by  the  MSI- 
Tracking  are: 

•  processing  of  input  data  from  different  types  of 
sensors  -  Primary  Radar,  IFF,  ESM,  Links 

•  merge  active  and  passive  tracking  (both  cooperative 
and  self  triangulation)  including  sophisticated 
correlation  and  association  logic.  This  makes  use  of 
geometric  data  and  attributes.  The  correlation  and 
association  logic  uses  mles  about  identity  indications 
from  ID  sources 

•  update  each  measurement  component  (Range,  AZ,  EL, 
Range  Rate)  separately,  using  a  special  form  of 
Kalman  Filter  algorithm 

•  automatically  associate  of  signature  and  attribute 
parameters  (IFF,  ESM,  ECM,  Link  data)  to  tracks 

•  update  attributes  (IFF  codes,  ESM  attributes,  ECM 
attributes) 

•  compensate  ownship  motion 

•  automatic  track  initiation  and  track  drop 

•  automatic  maneuver  detection 

•  measurement  dropout  coasting 

•  automatic  detection  &  compensation  of  bias 
(registration)  errors 

•  processing  of  operator  initiated  track  commands 

•  determination  of  target  environment  (air,  ground, 
surface)  based  on  kinematics  data 

Upon  reception  of  system  control  commands  issued 
automatically  by  the  MSI  monitoring  &  control  as  part  of 
MSI  management  SW,  the  tracking  adapts  its  functions 
automatically  to  graceful  degradation  measures  in  order  to 
prevent  uncontrolled  program  behavior  in  case  of  special 
conditions  (e.g.  overload).  Additionally,  the  MSI- 
Tracking  function  monitors  sensor  input  data  for 
plausibility  in  order  to  generate  inputs  for  error  logging. 

3.4  The  Multi-Sensor  Identification  System 

The  MSI  Identification  function  is  capable  to  identify  air, 
surface,  and  ground  tracks.  It  will  operate  fully  automatic 
and  uses  all  available  data  of  the  available  sensors  as  well 
as  derived  information  and  background  information 
dependent  on  the  confidence  which  can  be  given  to  the 
information  sources.  The  system  can  handle  different 
identification  schemes  in  parallel.  The  identification 
schema  is  loaded  as  data  during  system  initialization. 

High  flexibility  in  the  ID  process  is  achieved  by  providing 
the  operator  with  several  means  to  adapt  the  identification 
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function  to  mission  specific  needs.  Some  of  these  means 
are: 

•  modification  of  the  set  of  information  to  be  used  for 
identification^ 

•  modification  of  mission  data 

•  overriding  of  identification  results. 

These  functions  of  the  multi-sensor  identification  are 
based  upon  artificial  intelligence  concepts,  which  use  a 
rule  based  artificial  intelligence  system.  This  mle-based 
system  was  developed  by  DASA  ASD  with  the  focus  to 
be  used  in  operational  systems  and  give  responses  with 
minimum  delay. 

The  MSI  Identification  function  does  provide  the 
capability  to  assign  track  identities  based  on  integrated 
sensor,  communications  and  operator  provided  data.  ID 
related  information  as  for  example  conformance  of  tracks 
to  flight  plans,  or  platform  identification  by  origin 
(IDBO),  which  on  its  own  may  not  provide  conclusive  ID 
are  combined  in  order  to  extract  maximal  benefit  from  the 
available  sources  of  data.  Conflict  resolution  is  performed 
in  case  of  contradictory  data.  Therefore,  the  MSI- 
Identification  function  will  make  maximum  use  of  the 
available  information  to  provide  its  results. 

The  MSI  Identification  function  uses  the  information  from 
different  sources  to  determine  the  identity  of  MSI  tracks. 
These  sources  are: 

•  2D-position,  altitude,  speed  and  heading,  their  error 
values,  environment  category  (air,  surface,  ground) 
and  time  of  track  update  of  the  targets  from  MSI  Track 
File 

•  ID  related  data  from  local  AWACS-sensors  which  are 
attributes  of  MSI  Tracks 

•  ID  related  data  from  data  links 

•  Mission  data 

All  available  information  for  the  MSI  Identification 
process  is  used  in  a  most  effective  way  to  derive  a  unique 
target  identification  and  classification  for  the  environment 
categories  air,  surface,  and  ground.  The  identification  is 
done  automatically  or  by  operator  input.  The  manual  ID 
assigmnent  has  priority  over  automatically  determined  ID. 
In  case  of  manual  ID  assignment  automatic  ID 
determination  will  continue  and  MSI  ID  will  report 
detected  differences  between  automatic  identification 
based  on  available  information  and  the  manually  assigned 
ID. 

The  automatic  identification  process  will  evaluate  all 
available  identification  information  for  every  MSI  track 
using  „identification  indicators'*.  These  identification 
indicators  will  be  combined  into  a  track  identity  by  an 
artificial  intelligence  (AI)  supported  combination  process. 
Conflicts  in  the  available  identification  information  will 


be  detected.  The  automatic  system  will  provide  conflict 
resolution  functionality.  In  cases  where  conflict  resolution 
is  not  possible  operator  alerts  will  be  generated 
automatically.  Possible  ID  conflicts  with  ID  data  from  the 
data  links  are  indicated  to  the  operator.  In  case  of  detected 
severe  ID-changes  (e.g.  from  FRIEND  to  HOSTILE  or 
vice  versa)  a  manual  ID  assigmnent  will  be  requested. 

The  MSI  Identification  function  will  provide  rationale  for 
its  results  to  the  operator.  The  complete  MSI 
Identification  process  will  be  controlled  in  a  flexible  way 
by  operator  changeable  "Adaptation  Data". 

The  incoming  (live)  sensor  and  communication  data  may 
be  mixed  with  simulated  data  in  order  to  train  operators 
during  real  mission  flights. 

Upon  reception  of  system  control  commands  the  MSI 
Identification  adapts  its  functionality  automatically  to 
graceful  degradation  measures  to  prevent  uncontrolled 
program  behavior  in  case  of  special  conditions  (e.g. 
overload). 

3.5  MSI  Manager 

The  MSI  Manager  function  will  integrate  the  MSI 
Tracking  and  Identification  functions.  It  will  provide  all 
necessary  communication  mechanisms  for  MSI  Tracking 
and  MSI  Identification  based  on  the  DSI  middleware. 

The  MSICP  application  software  will  implement  the 
tracking  and  identification  functions  as  required  by  the 
MSICP  STRP.  It  will  make  use  of  the  DSI  and 
AMC/MSIC  System  Software  layers  and  provide  CORBA 
2.0  compliant  interfaces  for  external  communication  and 
communication  between  major  internal  software 
components. 

An  overall  MSICP  system  control  will  be  provided  by  the 
MSI  Manager  including  system  startup  and  shutdown 
mechanisms  as  well  as  the  overall  control  of  operational 
modes  of  the  MSICP.  The  MSI  Manager  will  also  provide 
an  overall  system  monitoring  function.  This  function  will 
provide  all  required  MSICP  system  status  data  in  regular 
manner  to  the  MSICP  external  interfaces. 

The  MSICP  will  be  responsible  for  redundancy  support  of 
MSI  Tracking  and  MSI  Identification  using  the  Boeing 
provided  checkpointing  service. 

The  MSI  Manager  will  be  responsible  for  the  overall 
computer  load  management  within  the  MSICP.  It  will 
measure  computer  load,  detect  overload  situations  and 
activate  the  load  adaptation  functionality  of  the  MSI 
Tracking  and  Identification  function  to  prevent 
unpredictable  behavior. 

Within  the  MSI  Manager  and  within  MSI  Tracking  and 
Identification  provisions  for  test  functionality  will  be 
included,  which  can  be  activated  by  the  MSI  Software 
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Test  Bed.  These  provisions  will  be  used  during  system 
testing  and  integration  and  may  be  used  for  maintenance 
activities  in  the  software  life  cycle.  These  provisions  will 
support: 

•  recording  of  external  and  internal  MSICP  interfaces 

•  logging  of  MSICP  internal  data  for  test  and 
maintenance  purposes. 

The  MSI  Software  Test  Bed  will  include  the  functionality 
to  make  use  of  these  provisions  including  the  external 
interfaces. 


4  SYSTEM  TEST 

During  Test  and  Development  the  MSI  Software  Test  Bed 
provided  by  DASA  Airborne  Systems  will  include  all 
functionality  needed  to  operate  the  MSICP  in  a  test 
environment,  without  having  the  Boeing  provided  AMCP 
available.  It  will  additionally  provide  simulations  of 
AMCP  functionality  as  needed  to  run  the  MSICP  in  a  test 
bed.  The  MSI  Software  Test  Bed  will  simulate  interfaces 
to  the  MSICP  compatible  to  the  AMCP  real  time 
interfaces.  Parts  of  this  MSI  Software  testbed  may  also  be 
used  for  maintenance  purposes  during  the  operational 
usage  of  the  MSICP.  Functionality  and  performance  of 
the  MSICP  will  be  verified  by  a  two  stage  approach.  First, 
testing  is  performed  via  computer  models.  The  Test  and 
Development  Support  software  contains  a  variety  of 
representative  scenarios  and  generates  the  inputs  of  all 
relevant  sources.  It  also  considers  the  sensor  behavior  that 
affects  detection  performance  and  accuracy  of  the 
measurements.  The  Test  and  Development  Support 
Software  (TADS)  will  include  all  features  needed  to 
generate  the  necessary  data,  and  do  the  analysis  of  results. 
Second,  the  MSICP  is  tested  by  using  recorded  life  data. 
The  evaluation  of  the  results  is  also  performed  by  the 
TADS. 

In  the  final  qualification  test  mn,  the  configuration  data  of 
the  multi-sensor  tracking  and  identification  function  will 
be  optimized  during  real  flight  tests.  This  will  finally 
verify  the  functions  and  the  performance  of  the  MSICP. 


Figure  2-2  Software  Layers  (Test  Configuration) 


5  PROGRAMME  STATUS 


After  subcontractor  selection  in  October  97  a  transition 
phase  was  initiated.  This  phase  prior  to  start  of 
Engineering  and  Manufacturing  Development  (EMD)  was 
used  for  requirements  clarification,  to  further  reduce 
program  risk  and  improve  program  affordability.  To  that 
end,  representatives  of  NAPMA,  the  user  community, 
MITRE,  US  Air  Force/Electronic  Systems  Center  (ESC), 
Boeing  and  ASD  worked  together  in  Integrated  Product 
Teams  (IPT). 

Engineering  and  Manufacturing  Development  started  in 
November  97  with  contract  award. 

The  first  version  of  the  MSICP  will  be  supplied  to  Boeing 
in  November  98  in  order  to  support  integration  testing. 
Further  software  versions  with  added  capabilities  will  be 
supplied  in  a  time  frame  of  roughly  6  months  until  the 
final  version  is  due  in  May  2000. 


6  SUMMARY 


In  this  paper,  a  brief  overview  of  the  NATO  AWACS 
Multi-Sensor  Integration  System  was  given.  It  explains 
the  requirements  for  the  new  system  and  its  performance 
drivers  as  well  as  the  concepts  which  were  applied  in  the 
system  design  in  order  to  match  the  requirements. 
Especially  in  the  identification  function  a  new  dimension 
of  flexibility  is  implemented  to  enable  operators  to  cope 
with  the  changing  real  world  situation. 
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Multi  Sensor  Integration  Computer  (MSIC) 
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Figure  3.3-2  Active  /  Passive  Correlation  and  Initiation 
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1.  SUMMARY 

Various  operational  trends  in  naval  warfare,  such  as 
technological  advances  in  threat  technology  and  an 
ongoing  shift  to  littoral  warfare,  put  the  shipboard 
decision  making  process  under  pressure.  Data  must  be 
processed  under  time-critical  conditions  and,  as  a 
consequence,  the  risk  of  saturation  in  building  the  tactical 
picture  and  of  making  the  wrong  decision  increases.  One 
must  also  realise  that  the  human  plays  an  essential  role  in 
the  naval  command  and  control  cycle.  Situation 
awareness  is  essential  for  commanders  and  their  staff  to 
conduct  decision-making  activities.  Data  Fusion  is  seen 
as  an  essential  process  to  enable  operators  to  achieve 
situation  awareness.  This  paper  discusses  the 
environment  perception  process  (EPP)  as  an  important 
aspect  of  the  problem  of  dynamic  decision  making  in  the 
context  of  naval  command  and  control.  The  EPP  is  aimed 
at  achieving  the  first  level  of  situation  awareness  that 
forms  the  basis  for  the  subsequent  more  abstract  levels. 
The  quality  of  the  results  of  the  EPP  is  of  utmost 
importance  for  the  situational  awareness  that  can  be 
achieved  at  the  higher  levels.  This  paper  is  a  step  towards 
the  definition  of  an  integrated  architecture  for  data  fusion 
giving  emphasis  to  situation  awareness  in  the  context  of 
dynamic  human  decision  making.  Such  an  integrated 
architecture  facilitates  the  proper  conceptualisation  and 
design  of  decision  support  systems  taking  into  account 
the  human  role  in  the  command  and  control  cycle. 

2.  INTRODUCTION 

At  the  heart  of  a  shipboard  combat  system  is  a  command 
and  control  system  (CCS)'  by  which  the  commanding 
officer  can  plan,  direct,  control  and  monitor  any 
operation  for  which  he  is  responsible,  to  defend  the  ship 
and  fulfil  his  mission.  The  increasing  tempo  and  diversity 
of  open-ocean  and  littoral  scenarios,  the  technological 
advances  in  threat  technology  and  the  volume  and 
imperfect  nature  of  the  data  to  be  processed  under  time- 


critical  conditions  pose  significant  challenges  for  future 
shipboard  CCS. 

This  emphasises  the  need  for  warships  to  be  fitted  with  an 
efficient  combat  system  featuring  a  real-time,  joint 
human-machine  decision  support  system  (DSS)  integrated 
into  the  ship’s  CCS.  This  DSS  consists  in  the 
combination  of  a  multi-source  data  fusion  (MSDF) 
capability,  a  situation  and  threat  assessment  (STA) 
capability,  and  a  resource  management  (RM)  capability 
(including  the  management  of  weapons,  sensors, 
navigation  and  communication).  These  capabilities 
intimately  match  the  four  levels  of  the  JDL  (Joint 
Directors  of  Laboratories)  data  fusion  model.  The  main 
role  of  such  a  real-time  DSS  is  to  aid  the  ship’s  operators 
to  achieve  the  appropriate  situation  awareness  (SA)  state 
for  their  tactical  decision-making  activities,  and  to 
support  the  execution  of  the  resulting  actions. 

The  Decision  Support  Technologies  Section  at  the 
Defence  Research  Establishment  Valcartier  (DREV, 
Canada)  and  the  Maritime  Command  and  Control  group 
of  the  Physics  and  Electronics  Laboratory  of  the 
Netherlands  Organisation  of  Applied  Scientific  Research 
(TNO-FEL,  The  Netherlands)  are  both  conducting 
research  and  development  (R&D)  activities  in  the  field  of 
decision  support  for  Maritime  Command  and  Control  at 
the  shipboard  level.  Investigations  have  been  undertaken 
to  study  the  concepts  and  design  of  a  real-time  DSS  for 
their  respective  frigates  in  order  to  improve  their 
performance  against  current  and  future  threats. 

This  paper  presents  a  brief  overview  of  one  particular 
collaborative  effort  focusing  at  deriving  a  functional 
decomposition  of  the  data  fusion  process  giving  emphasis 
to  situation  awareness  concepts  [Ref.  1].  This  is 
important  for  the  integration  of  the  human  element  into 
the  design  of  an  efficient  decision  support  system  aiding 
the  operators  to  achieve  the  appropriate  situation 
awareness  (SA).  SA  is  essential  for  commanders  and  their 
staff  to  conduct  decision-making  activities.  Data  Fusion 
(DF)  is  seen  as  an  essential  process  to  enable  operators  to 
achieve  situation  awareness. 


Atso  known  as  the  Combat  Direction  System  (CDS) 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-I2. 
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The  paper  is  thus  a  step  towards  the  definition  of  an 
integrated  data  fusion  architecture  adapted  to  human’s 
needs  for  situation  awareness  and  dynamic  decision 
making  in  maritime  command  and  control.  The 
eiiviroiunent  perception  process  (EPP)  is  part  of  this 
architecture  and  is  aimed  at  achieving  the  first  level  of 
situation  awareness.  This  first  level  forms  the  basis  for 
the  other,  more  abstract  levels.  Thus,  the  quality  of  the 
results  of  this  level  is  of  utmost  importance  for  the 
situational  awareness  that  can  be  achieved  at  the  higher 
levels.  The  EPP  is  analysed  in  this  paper,  and  the 
input.s/outputs  of  its  sub-processes  are  identified.  With  a 
better  knowledge  of  the  environment  perception  process, 
one  can  proceed  with  the  study  of  the  processes  required 
to  achieve  the  higher  levels  of  SA. 

This  paper  is  organised  as  follows.  Section  3  provides 
background  information  on  the  command  and  control 
(C2)  process  and  the  role  of  a  decision  support  system  in 
this  process.  Data  fusion  and  the  role  of  situation 
awareness  in  dynamic  human  decision-making  are  also 
presented  in  this  section.  Making  a  link  between  multiple 
models,  section  4  then  discusses  data  fusion  from  a 
situation  awareness  perspective.  A  functional 
decomposition  of  the  environment  perception  process  is 
presented  and  described  in  section  5.  Section  6  briefly 
discusses  potential  pitfalls  in  designing  an  integrated 
architecture  for  data  fusion.  Section  7  provides 
conclusions  and  recommendations,  and  discusses  future 
work. 

3.  BACKGROUND 

3.1  Command  and  Control  Process 

Command  and  control  (C2)  is  the  process  by  which 
commanders  can  plan,  direct,  control  and  monitor  any 
operation  for  which  they  are  responsible  [Ref.  2].  In  a 
naval  context,  most  tactical  decisions  taken  within  the 
ship’s  operations  room  are  made  after  completing  a 
number  of  perceptual  procedural  and  cognitive  activities 
linked  to  the  C2  process.  The  C2  process  is  indeed  a  suite 
of  periodic  activities  which  mainly  involves  the 
perception  of  the  domain  (environment),  an  assessment  of 
the  tactical  situation,  decision  making  about  a  course  of 
action  and  the  implementation  of  the  chosen  plan 
The  C2  activities  are  performed  by  either  humans, 
machines  (i.e.,  hardware  and  software  computer  systems), 
or  a  combination  of  both.  Characteristics  of  this  suite  of 
activities  are  described  in  Ref.  3  and  were  captured 
through  the  Boyd’s  Observe-Orient-Decide-Act  (OODA) 
loop  illustrated  in  Figure  1.  Although  this  loop  might  give 


the  impression  that  C2  processes  are  executed  in  a 
sequential  way,  in  reality,  the  processes  are  concurrent 
and  hierarchically  structured. 


Figure  1  -  Boyd's  OODA  loop. 

The  military  community  typically  states  that  the  dominant 
requirement  to  counter  the  threat  and  ensure  the 
survivability  of  the  ship  is  the  ability  to  perform  the  C2 
activities  (i.e.,  the  OODA  loop)  quicker  and  better  than 
the  adversary.  Therefore,  the  speed  of  execution  of  the 
OODA  loop  and  the  degree  of  efficiency  of  its  execution 
are  the  keys  of  success  for  shipboard  tactical  operations. 
Decision  support  systems  can  contribute  significantly  to 
the  execution  of  this  loop. 

3.2  Decision  Support  System 

The  complexity  of  the  shipboard  environment  in  which 
operators  conduct  C2  activities  emphasises  the  need  for 
warships  to  be  fitted  with  a  real-time  decision  support 
system  (DSS).  The  main  role  of  this  DSS  is  to  aid  the 
operators  in  achieving  the  appropriate  situation  awareness 
in  order  to  support  them  in  their  tactical  decision  making 
and  action  execution  activities. 

Operators  need  to  be  aided  by  a  DSS  that  continuously 
fuses  data  from  the  ship’s  sensors  and  other  sources 
(MSDF  capability),  helps  the  operators  maintain  a  picture 
of  the  tactical  situation  (STA  capability),  and  supports 
their  response  to  actual  or  anticipated  threats  (RM 
capability). 

Figure  2  presents  the  mapping  of  the  MSDF/STA/RM 
system  onto  the  OODA  loop.  The  data  fusion  process, 
described  in  the  next  section,  is  seen  as  an  important 
element  of  a  DSS  to  provide  the  appropriate  situation 
awareness  to  operators  in  support  of  their  C2  activities. 
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Figure  2  -  Mapping  of  the  MSDF/STA/RM  system 
onto  the  OODA  loop. 

3.3  Data  Fusion 

According  to  the  JDL  model  (see  Refs.  4-15),  DF  is 
fundamentally  a  process  designed  to  manage,  organise, 
combine  and  interpret  data  and  information  obtained 
from  a  variety  of  sources,  that  may  be  required  at  any 
time  by  operators  and  commanders  for  decision  making. 
It’s  an  adaptive  information  process  that  continuously 
transforms  the  available  data  and  information  into  richer 
information.  Refined  (and  potentially  optimal)  kinematics 
and  identity  estimates  of  individual  objects,  and  complete 
and  timely  assessments  of  current  and  potential  future 
situations  and  threats  (i.e.,  contextual  reasoning)  are 
achieved  through  continuous  refinement  of  hypotheses  or 
inferences  about  real-world  events.  The  DF  process  is 
also  characterised  by  the  evaluation  of  the  need  for 
additional  data  and  information  sources,  or  the 
modification  of  the  process  itself,  to  achieve  improved 
results. 

Given  these  considerations,  a  complete  DF  system  can 
typically  be  decomposed  into  five  levels: 

•  Level  0  -  Signal  Data  Refinement  (  source  pre¬ 
processing); 

•  Level  1  -  Object  Refinement  (Multi-Source  Data 
Fusion  (MSDF)); 

•  Level  2  -  Situation  Assessment  (SA); 

•  Level  3  -  Threat  Assessment  (TA);  and, 

•  Level  4  -  Process  Refinement  through  Resource 
Management  (RM). 


Each  succeeding  level  of  DF  processing  deals  with  a 
higher  level  of  abstraction.  Level  1  DF  uses  mostly 
numerical,  statistical  analysis  methods,  while  levels  2,  3, 
and  4  of  DF  use  mostly  symbolic  or  Artificial  Intelligence 
(AI)  methods.  Note  that  resource  management  in  the 
context  of  level  4  fusion  is  mainly  concerned  with  the 
refinement  of  the  information  gathering  process  (e.g., 
sensor  management).  However,  the  overall  domain  of 
resource  management  also  encompasses  the  management 
of  weapon  systems  and  other  resources  (including  the 
management  of  navigation  and  communication  systems). 
The  JDL  model  provides  a  good  description  of  the  data 
fusion  process.  This  process  is  an  important  element 
within  the  C2  cycle.  One  must  also  realise  that  the  human 
plays  an  essential  role  in  the  C2  cycle.  He  is  the  one 
responsible  for  taking  decisions  [Refs.  16-17].  Because  of 
the  importance  of  humans,  one  needs  a  mechanism  to 
reason  about  their  role  in  the  C2  cycle  in  order  to 
facilitate  the  proper  conceptualisation  and  design  of  DSS. 
Endsley  [Ref  18]  showed  that  situation  awareness  is  an 
essential  precondition  in  this  decision  making  process. 

3.4  Situation  Awareness 

Endsley  has  derived  a  theoretical  model  of  situation 
awareness  based  on  its  role  in  dynamic  human  decision 
making.  Endsley  [Ref  1 8]  defines  situation  awareness  as 
the  perception  of  the  elements  in  the  environment  within 
a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near 
future.  Figure  3  depicts  the  three  levels  of  situation 
awareness  as  identified  by  Endsley. 

SA  can  be  interpreted  as  the  operator’s  mental  model  of 
all  pertinent  aspects  of  the  environment  (process,  state, 
and  relationships).  This  mental  model  of  the  environment 
is  also  known  in  Rasmussen’s  work  as  the  hierarchical 
knowledge  representation  (HKR)  in  decision-making 
[Refs.  19-21].  Although  this  paper  focus  at  the  process 
building  this  HKR,  R&D  works  is  ongoing  to  define  in 
details  this  HKR  in  the  context  of  naval  C2. 

Finally,  bearing  in  mind  the  scope  of  this  paper,  one 
should  note  that  SA  could  be  achieved  without  the 
transformation  and  the  fusion  of  data.  For  instance, 
training  techniques  typically  enhance  the  operator’s 
performance,  resulting  in  a  better  SA.  Similarly, 
advanced  techniques  in  human  computer  interaction 
(HCI)  allow  a  representation  of  the  information  in  a 
meaningful  way  for  the  human. 
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Figure  3  -  SA  model  in  dynamic  human  decision  making 


4.  SA  PERSPECTIVE  FOR  DATA  FUSION 

If  one  compares  the  OODA  loop  with  the  SA  model  of 
Endsley,  one  sees  a  close  resemblance.  In  both  models 
one  finds  a  decision-making  part  and  an  action  part.  In 
Endsley's  model,  SA  is  one  of  the  main  inputs  for 
decision-making.  In  the  OODA  loop,  the  processes 
Observe  and  Orient  provide  inputs  for  the  decision 
making  process.  One  should  recall,  however,  that 
situation  awareness  in  Endsley's  model  is  a  state  of 
knowledge  and  not  a  process.  The  processes  Observe 
(MSDF)-  and  Orient  (STA)  of  the  OODA  loop  can  thus 
be  seen  as  processes  acquiring,  achieving,  and 
maintaining  situation  awareness.  In  turn,  this  means  that 
we  can  link  the  observation  and  orientation  processes  to 
the  situation  awareness  model  of  Endsley. 

The  MSDF  process  (or  level  0-1  fusion)  can  be  viewed  as 
the  process  providing  the  first  level  of  SA.  It  is 
responsible  for  the  perception  of  the  status,  attributes, 
and  dynamics  of  relevant  elements  in  the  environment.  If 
we  look  at  our  problem  domain,  maritime  command  and 
control,  the  basic  element  relevant  for  the  command  team 
is  any  object  in  the  environment  (e.g.,  air,  surface  or 
subsurface  target). 


The  STA  process  is  responsible  in  providing  the  next  two 
levels  of  SA.  The  second  step  in  achieving  situation 
awareness  is  called  comprehension.  Endsley  describes  the 
comprehension  process  as  follows:  "Comprehension  of 
the  situation  is  based  on  a  synthesis  of  disjointed  level  1 
elements".  Level  2  of  situation  awareness  goes  beyond 
simply  being  aware  of  the  elements  that  are  present  to 
include  an  understanding  of  the  significance  of  those 
elements  in  light  of  pertinent  operator  goals.  Based  on 
knowledge  of  Level  1  elements,  particularly  when  some 
elements  are  put  together  to  form  patterns  with  other 
elements,  the  decision-maker  forms  a  holistic  picture  of 
the  environment,  comprehending  the  significance  of 
objects  and  events. 

The  third  and  last  step  in  achieving  situation  awareness  is 
the  projection  of  the  future  actions  of  the  elements  in  the 
environment.  This  is  achieved  through  knowledge  of  the 
status  and  dynamics  of  the  perceived  and  comprehended 
situation  elements. 

In  order  to  bootstrap  the  functional  decomposition  of  an 
integrated  data  fusion  architecture  taking  into  account 
Endsley’s  SA  theory,  the  processes  leading  to  the  first 
level  of  SA  are  analysed  and  discussed  in  the  next 
section. 


•  Here  level  0  is  included  in  MSDF 
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5.  ENVIRONMENT  PERCEPTION  PROCESS 

As  mentioned  above,  the  perception  process  leading  to 
the  first  level  of  SA  corresponds  to  levels  0-1  of  the  data 
fusion  model.  An  important  characteristic  of  this  data 
fusion  model  is  that  it  is  an  adaptive  and  iterative  multi¬ 
level  process  that  continuously  transforms  the  available 
data  into  richer  information.  One  should  also  note  that 
these  levels  of  fusion  are  linked  together  and  cannot  be 
considered  independently  without  missing  functionalities 
and/or  reducing  the  quality  of  their  results.  The  functional 
decomposition  of  the  perception  process  and  the  quality 
of  its  results  (the  estimated  perception  of  the 
environment)  are  thus  different  whether  the  process  is 
implemented  as  an  opened  or  a  closed  loop. 

The  closed  loop  implementation  inherently  uses  the 
notion  of  process  refinement  (explained  later  in  this 
section).  This  means  that  the  perception  process  will  be 
refined  and  enhanced  leveraging  from  the  results  of 
higher  levels  of  data  fusion.  As  a  result,  the  perception 
process  then  benefits  indirectly  of  contextual  information. 
This  is  a  major  difference  from  the  opened  loop 
implementation  where  the  results  of  the  perception 
process  are  context-free. 

In  view  of  the  discussion  above,  the  environment 
perception  process  (EPP)  is  referred  to  as  the  process 
achieving  the  first  level  of  situation  awareness  according 
to  Endsley’s  theory,  through  a  closed  loop 
implementation. 

The  main  requirement  for  a  closed  loop  implementation 
and  for  the  use  of  the  process  refinement  notion  is  the 
integration  of  all  levels  of  data  fusion.  Unfortunately, 
most  of  the  R&D  efforts  in  data  fusion  have  been  in  the 
domain  of  opened  loop  levels  0-1  fusion.  Accordingly, 
this  domain  is  more  mature  and  years  of  R&D  have  lead 
to  the  definition  of  a  generic  MSDF  system  [Ref.  1 1  ] 
shown  in  Fig.  4. 


Figure  4  -  Generic  MSDF  System 


An  MSDF  system  processes  the  information  data  reported 
by  multiple  dissimilar  sources  in  order  to  correctly  and 


quickly  derive  the  best  estimates  of  the  current  and  future 
kinematics  properties  for  each  hypothesised  (or 
perceived)  entity  in  the  operational  environrhent,  and  to 
develop  inferences  as  to  the  identity  and  key  attributes  of 
these  entities. 

The  MSDF  system  attempts  to  acquire  and  maintain 
unambiguous,  stable  tracks  corresponding  to  the 
perceived  population  of  real  objects  within  the 
operational  volume  of  interest  (i.e.,  establish  a  number  of 
clean  tracks  that  corresponds  exactly  to  the  number  of 
objects  in  the  physical  environment). 

The  functional  decomposition  of  the  EPP  is  derived 
leveraging  from  this  definition  of  a  generic  MSDF  system 
that  was  itself  derived  without  explicitly  considering  the 
results  of  higher  levels  of  fusion  processing.  The 
following  section  describes  the  extended  notion  of 
process  refinement  including  the  use  of  the  results  from 
higher  levels  of  processing. 

5.1  Process  Refinement  in  Data  Fusion 

One  can  distinguish  three  different  types  of  process 
refinement  in  a  multi-level  data  fusion  process.  First, 
providing  information  with  less  uncertainty  at  its  input 
can  refine  a  process.  In  this  case,  refined  inputs  leads  to 
enhanced  results  of  the  process.  This  type  of  refinement 
can  be  achieved  by  enhancing  the  results  of  a  prior 
stovepipe  process,  the  results  of  a  lower  level  process  or, 
the  environment  sensing  process. 

Another  way  to  refine  a  process,  without  changing  the  set 
of  inputs,  is  to  modify  the  algorithm  used  for  the  process 
by  either  tuning  its  parameters  or  selecting  a  totally 
different  algorithm.  This  type  of  refinement  is  referred  to 
as  process  control. 

Additional  or  complementary  processing  based  on  the 
results  of  higher  levels  of  fusion  processing  corresponds 
to  the  third  type  of  process  refinement.  This  type  of 
refinement  has  not  been  considered  in  the  previous 
derivation  of  the  generic  MSDF  system. 

5.2  Examples 

By  illustrating  the  refinement  mechanisms  shortly 
described  in  the  previous  subsection  with  two  examples, 
we  show  that  these  mechanisms  are  beneficial. 

The  first  example,  illustrate  in  Figure  5,  is  about  the 
provision  of  extra  information  regarding  the  possible 
existence  of  a  target  currently  not  detected.  Suppose  that 
a  number  of  air  targets  have  been  observed,  proceeding 
close  to  each  other,  with  the  same  course  and  speed.  If 
the  inferred  identity  of  the  targets  and  the  structure  of  the 
formation  are  combined  with  our  knowledge  of  doctrines, 
we  can  infer  that  it  is  likely  a  formation  of  a  specific  type. 
According  to  our  knowledge  of  doctrines  however,  a 
formation  of  the  inferred  type  usually  consists  of  four 
targets,  while  only  three  targets  have  been  observed. 
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Figure  5  -  Example  1:  formation  fighters 

Knowing  all  this,  the  results  of  the  EPP  can  be  enhanced 
by  providing  the  process  with  this  extra  information.  For 
instance,  creating  a  new  track  hypothesis  could  do  this.  In 
turn,  the  existence  of  this  extra  hypothesis  may  have 
impact  on  several  sub-processes  of  the  EPP  (data 
association,  cluster  management,  etc.).  Furthermore,  it 
may  even  have  impact  at  the  sensor  level.  A  radar  system 
might  be  cued,  for  example,  to  perform  some  extra 
volume  search  dwells  to  verify  the  new  track  hypothesis. 
As  a  variant  of  this  example,  one  could  also  imagine  that 
the  fourth  target  has  already  been  detected,  but  not 
identified.  In  this  case,  the  EPP  could  be  provided  with 
evidence  for  the  type  of  the  target. 


Figure  6  -  Example  2:  bearing-only  track 

The  second  example  (Figure  6)  is  about  a  request  tor 
extra  information  about  the  kinematics  of  a  track. 
Suppose  there  is  a  bearing-only  track  based  on  ESM 
measurements  of  the  seeker-head  of  an  incoming  missile. 


For  effective  engagement  of  the  missile  with  hard-kill  and 
soft-kill  measures,  information  about  the  distance  ot  the 
target  to  our  own  ship  is  of  relevance.  Some  distance 
information  can  often  be  derived  from  the  mode  of  the 
seeker-head  of  the  missile  (searching,  lock-on).  However, 
more  precise  distance  information  can  be  of  help. 

In  the  situation  described  above,  an  information  request 
can  be  issued.  It  is  unlikely  that  this  request  will  have  an 
important  impact  on  the  EPP  itself.  If  distance 
information  would  have  already  been  available  somehow, 
it  should  have  been  associated  with  the  bearing  track.  It  is 
more  likely  for  such  an  information  request  to  have  an 
impact  on  sensors.  Like  for  the  previous  example,  a  radar 
might  be  cued  to  perform  a  search  dwell  to  find  out 
whether  a  radar  track  can  be  initiated  that  can  be 
associated  with  the  existing  bearing-only  track. 

5.3  Environment  Perception  Process  Framework 
Figure  7  is  an  attempt  to  illustrate  the  environment 
perception  process.  The  EPP  framework  is  composed  of 
the  levels  0-1  functional  decomposition  as  if  they  were  in 
an  opened  loop.  As  explained  earlier,  the  EPP  is  the 
perception  process  within  a  closed-loop  using  the  notion 
of  process  refinement.  This  can  be  visualised  in  Figure  7 
by  the  presence  of  the  level  4  fusion,  the  additional 
processes  for  levels  0  and  I ,  and  the  outputs  of  higher 
levels  of  fusion  redirected  toward  levels  0- 1 . 


Figure  7  -  EPP  Framework 


5.4  Tactical  Information  Abstraction  Framework 

In  her  theory  of  situation  awareness,  Endsley  clearly 
presumes  patterns  and  higher  level  elements  to  be  present 
according  to  which  the  situation  can  be  structured  and 
expressed.  In  Section  3.4  we  already  made  a  link  between 
Endsley’ s  situation  awareness  model  and  the  work  of 
Rasmussen.  In  the  context  of  our  problem  domain, 
maritime  command  and  control,  we  would  call  such  a 
structuring  language  Tactical  Information  Abstraction 
Framework  (TIAF). 

If  we  are  able  to  use  a  TIAF  in  the  data  fusion  process 
according  to  which  situation  awareness  can  be  structured, 
we  can  make  a  smooth  match  of  this  process  with  the 
situation  awareness  framework.  Thus  we  will  be  able  to 
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integrate  the  human  element  into  the  design  of  a  decision 
support  system  aiding  the  operators  to  achieve  the 
appropriate  situation  awareness. 

6.  UNCERTAINTY  AND  POTENTIAL  PITFALLS 
In  the  context  of  naval  command  and  control,  the 
imperfect  nature  of  the  result  of  any  data  fusion  process  is 
caused  by  several  reasons.  First  of  all,  the  sources  of 
information  provide  incomplete,  noisy,  and  inconsistent 
data  because  of  the  physical  and/or  environmental 
constraints.  Furthermore,  due  to  restrictions  imposed  by 
the  mission,  use  of  active  sensors  (such  as  radar  systems) 
is  not  always  allowed.  Uncertainty  can  be  connected  to 
the  existence  of  a  track.  This  type  of  uncertainty  is  often 
expressed  in  terms  like  pending,  tentative  and  confirmed. 
Uncertainty  can  also  be  connected  with  the  kinematics 
attributes  of  a  track.  In  our  second  example  we  saw  a 
target  of  which  only  the  bearing  was  known.  But  even  if 
the  position  and  the  speed  of  a  track  are  known,  there  is 
likely  to  be  uncertainty  associated  with  their 
measurements  or  estimates.  Uncertainty  can  also  be 
connected  with  the  identity  of  a  track.  What  type  or  class 
of  target  the  track  represents  can  be  represented  with 
identity  propositions. 

The  above-mentioned  uncertainty  types  are  related  to 
tracks.  There  is  also  uncertainty  of  a  different  kind.  It 
might  be  the  case  that  a  particular  .sensor  has  not  covered 
a  certain  geographical  or  spatial  area.  There  are  several 
realistic  reasons  for  this.  The  mission  of  our  platform 
might  imply  restrictive  use  of  active  sensors.  A  spatial 
area  may  be  obscured  by  a  big  object  or  land  mass.  A 
more  obvious  reason  would  be  a  restriction  ot  the  sensor 
range  due  to  physical  factors  (such  as  radar  horizon). 

For  an  effective  use  of  the  information  request  and 
provision  mechanism,  it  is  important  that  all  these  types 
of  uncertainty  are  explicitly  represented.  Only  if  this  is 
the  case,  extra  information  can  be  provided  and/or 
requested  to  the  point.  A  serious  complication  is  that  all 
these  different  kinds  of  uncertainty  will  not  likely  be 
expressed  in  a  similar  uncertainty  paradigm.  Because  of 
the  multi-type  uncertainty  problem,  many  uncertainty 
conversions  may  be  necessary  [Refs.  22-25]. 

Another  potential  problem  is  as  follows.  If  new 
information  is  derived  from  the  information  received 
from  the  lower  data  fusion  levels  and  if  this  new 
information  is  later  provided  to  the  lower  levels  as  an 
extra  information  source,  we  must  be  aware  of  a  serious 
risk  of  data  corruption  (e.g.,  data  looping)  [Ref.  26].  The 
information  provided  to  the  lower  levels  will  somehow  be 
worked  into  the  product  we  receive  from  these  lower 
levels.  We  somehow  must  make  provisions  in  order  to 
prevent  such  data  looping.  We  will  illustrate  this  risk  with 
the  first  process  refinement  example  we  gave.  Based  on 
the  ob.servation  of  three  fighters  of  a  certain  type,  we 


posed  a  hypothesis  of  a  specific  formation.  From  this 
hypothesis,  we  derived  evidence  for  a  fourth  fighter  at  a 
specific  position  in  the  formation.  If  we  provide  this  extra 
information  to  the  EPP  we  will  receive  this  hypothesis 
from  EPP,  as  a  track  hypothesis.  As  long  as  the  track 
hypothesis  has  not  been  confirmed  by  real  observations, 
or  by  other  means,  we  must  be  aware  of  the  fact  that  we 
cannot  use  this  track  hypothesis  as  extra  evidence  for  the 
formation  hypothesis. 

7.  CONCLUSION 

This  paper  presented  the  first  step  toward  an  integrated 
architecture  for  data  fusion  process  giving  emphasis  to 
situation  awareness  concepts.  The  main  objective  is  the 
integration  of  the  human  element  into  the 
conceptualisation  and  the  design  of  efficient  computer- 
based  tools  adapted  to  human’s  needs  for  situation 
awareness  and  dynamic  decision  making  in  maritime 
command  and  control. 

The  paper  has  highlighted  the  need  to  study,  in  a  closed 
loop  and  integrated  environment,  the  perception  process 
according  to  SA  concepts.  The  resulting  model  coupled 
with  a  Tactical  Information  Abstraction  Framework  will 
allow  to  structure  and  express  the  knowledge  or 
information  in  a  meaningful  way  to  the  human 
responsible  to  take  a  decision. 

Future  work  is  planned  to  broaden  the  scope  of  this  study 
for  the  integration  of  all  data  fusion  levels  and  also  to 
assess  the  use  of  blackboard  systems  as  an  integration 
framework  for  the  data  fusion  process.  Finally,  efforts  are 
also  planned  to  derive,  using  Rasmussen’s  methodology, 
the  tactical  information  abstraction  framework  needed  to 
structure  and  express  the  information  to  the  decision¬ 
maker. 
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Abstract 

A  concept  for  a  new  flight  guidance  system  that 
focuses  on  the  problem  of  low  level  flight  and 
reduction  of  crew  workload  for  a  military  transport 
aircraft  is  presented.  A  digital  terrain  database  is 
used  to  eliminate  the  need  for  an  active,  forward- 
looking  radar,  thus  permitting  silent  terrain 
following  and  terrain  avoidance.  Coupled  to  the 
database  are  a  highly  reliable,  precision 
navigation,  4D  flight  guidance  and  display 
functions.  The  demonstration  of  key  technologies 
associated  with  this  system  has  been  carried  out 
in  several  R&D  programmes  to  prove  the  high 
maturity  of  available  technologies  and  to  reduce 
the  development  risks.  The  target  aircraft  for 
these  studies  is  the  future  European  tactical 
transport  aircraft,  known  as  Future  Large  Aircraft 
(FLA)  or  Future  Transport  Aircraft  (FTA).  The 
present  low  level  flight  (LLF)  technical  solution  has 
been  prototyped  and  tested  by  German  Air  Force 
in  Airbus  Experimental  Cockpit  Simulator  and  two 
flying  testbeds  (Cl  60  Transall  and  ATTAS).  The 
experimental  verification  process  is  still  currently 
in  progress. 


Introduction 

The  development  of  a  modern  flight  guidance 
system  for  military  transport  aircraft  must  meet 
the  requirements  of  future  mission  scenarios.  The 
mission  success  of  many  of  these  scenarios  is 
greatly  increased  through  the  use  of  low-level  and 
terrain  following  flight,  where  the  terrain  can  be 
effectively  used  to  mask  the  aircraft  from  enemy 
threats.  Typical  tactical  mission  profiles  can 
include  long  distance  low-level  segments  of  up  to 
500  NM  or  more.  Additionally,  operations  must  be 
conducted  from  any  main  base  in  adverse  weather 
conditions  and,  at  times,  from  forward  operating 


strips  in  increasingly  sophisticated  and  hostile 
environment. 

Low  altitude,  terrain  following  flight  for  large 
transport  aircraft  is  a  challenge  that  bears  some 
similarities,  but  many  differences,  from  the 
ground-hugging  flight  of  an  attack  aircraft.  Military 
transports  obviously  have  more  structural 
constraints,  but  the  biggest  difference  is  a  low 
thrust-to-weight  ratio,  especially  when  heavily 
loaded.  Lacking  a  tactical  aircraft's  ability  to  pop 
up  quickly  to  follow  the  terrain,  a  transport's  pull- 
ups  over  rising  terrain  must  be  planned  much 
further  ahead,  which  requires  a  detailed 
knowledge  of  the  terrain  ahead. 

During  terrain  following,  both  the  crew  and  the 
aircraft  are  pushed  to  the  limits.  Thus  a  system, 
that  supports  and  aids  the  crew  in  this  difficult 
task,  is  required.  For  low  level  flight  to  be 
effective,  the  system  must  be  silent  and  not  use 
any  high-powered,  detectable  sensors,  such  as 
forward  looking  radar.  The  use  of  digital  terrain 
data  can  effectively  fulfil  this  requirement  and 
provide  maximum  safety. 

Therefore  military  transport  aircraft  must  be 
capable  of  flying  autonomously  and  automatically 
at  the  lowest  possible  altitude,  where  the  crew  is 
supported  by  a  four  dimensional  low-level  flight 
guidance  system.  Therefore,  the  system  must  be 
able  to  perform  combined  vertical  and  horizontal 
terrain  avoidance/terrain  following  manoeuvres, 
which  assists  the  aircrew  by  reducing  their 
workload  during  a  low-level  flight  phase  within  a 
dynamic  threat  environment,  or  at  the  presence  of 
a  system  failure.  The  system  can  also  increase  the 
effectiveness  of  logistic  missions,  where  flying  in 
poor  weather  forces  the  pilot  to  fly  under  the 
cloud  level  and  close  to  the  ground  to  drop 
payloads  under  line-of-sight  conditions. 


Paper  presented  at  the  RTO  SCI  Symposium  on  “Sensor  Data  Fusion  and  Integration  of  the  Human  Element”, 
held  in  Ottawa,  Canada,  14-17  September  1998,  and  published  in  RTO  MP-12. 
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In  this  paper,  a  new  system  concept  developed  by 
DaimlerChrysler  Aerospace  (Dasa)  will  be 
presented.  The  main  components  of  the  system 
include  a  digital  terrain  database,  an  integrated, 
precision  navigation  system,  and  4-dimensional 
flight  guidance  and  display  functions.  The  heart  of 
the  system  is  a  digital  terrain  model  used  for 
autonomous,  jam-resistant,  terrain-referenced 
navigation  and  the  generation  of  4-dimensional 
flight  trajectories  (Figure  1).  Relevant  information 
is  presented  to  the  crew  on  a  combination  of  Head 
Up,  Head  Down  and  Navigation  Displays  for 
improved  situation  awareness.  The  system  has 
been  prototyped  and  individual  key  technologies 
developed  and  demonstrated  in  three  key  research 
and  development  programmes.  The  demonstration 
of  system  components  within  each  technology 
programme  has  been  accomplished  with  German 
Air  Force  transport  pilots  in  two  flying  testbeds 
and  in  the  FLA  Experimental  Cockpit  Simulator  at 
Deutsche  Airbus  in  Hamburg,  Germany. 


Experience 

Dasa  (formerly  Messerschmidt-Bolkow-Blohm  and 
Vereinigte  Flugtechnische  Werke)  has  nearly  two 
decades  of  Terrain  Following  (TF)  experience, 
starting  with  the  System  Design  Responsibility 
(SDR)  for  the  Tornado's  TF  system  and 
culminating  in  1993  with  the  Tornado  flight 
demonstration  of  LATAN  (Low  Altitude  Terrain 
Avoidance  and  Navigation).  LATAN,  a  terrain 
databased  navigation  and  terrain  following 
system,  completed  more  than  40  successful 
sorties  and  the  entire  Tornado  low-level  flight 
envelope  under  LATAN  control  was  tested.  Due  to 
the  system's  maturity,  LATAN  was  granted  the 
experimental  flight  clearance  by  the  German 
Federal  Office  for  Military  Technology  and 
Procurement  (BWB-ML)  to  fly  as  a  stand  alone 
system  in  the  automatic  LATAN  TF  mode  down  to 
200  feet  SCH. 


System  Design 

The  system  designed  by  Dasa  for  low  level  and 
terrain  following  flight  is  built  around  a  central 
digital  terrain  and  mission  database.  This  database 
is  critical  to  the  system's  functional  process  and 
flow  of  information.  Therefore  the  entire 
functional  LLF  process,  especially  the  navigation 
function,  must  be  treated  as  flight  safety  critical 
(Figure  2). 

The  key  system  components  and  features  are  as 
follows: 

•  Central  Mission  Database:  The  database  is 
comprised  of  all  terrain  and  mission  data 
required  to  successfully  complete  a  tactical 


and/or  logistical  mission.  As  such,  it  supplies 
the  navigation,  flight  planning,  4D  flight 
guidance,  and  display  processes  with  all 
relevant  data  (Figure  3).  The  terrain  data  is  a 
fusion  of  various  data  sources  (DTED,  DEAD, 
LFH,  satellite  photos,  radar  imagery,  etc.).  The 
pre-processing  procedure  must  verify  the  data 
before  it  is  loaded  into  the  onboard  sy 
stem. 

A  highly  sophisticated  navigation  system  based 
on  the  sensor  fusion  of  laser  inertial  navigation 
system  (LINS),  global  positioning  system  (GPS) 
with  military  P/Y  code,  and  terrain  referenced 
navigation  (TRN).  By  using  the  strengths  of 
each  individual  sensor,  for  example,  the  GPS's 
precision,  the  weaknesses  can  be  overcome  to 
provide  a  navigation  solution  that  has  a  higher 
reliability,  integrity  and  availability  than  any 
single  sensor.  The  fusion  is  accomplished 
through  the  use  of  4  stage  extended  Kalman 
Filter  (1  primary  filter  for  all  three  sensors  and 
3  secondary  filters  for  each  pair)  and  an 
associated  failure  detection  and  isolation  (FDI) 
logic  for  the  switching  between  filters.  The 
result  is  a  precise  and  reliable  navigation  vector 
that  can  be  used  for  safety  critical  applications. 
The  LLF  system  has  a  Care  Free  Moding 
(Figure  4)  to  provide  the  aircrew  with 
maximum  flexibility  through  various  levels  of 
automatic  support.  The  aircrew  is  able  to 
switch  between  modes  at  will.  The  switching 
between  or  coupling  to  TF  routes  is  always 
accomplished  with  full  terrain  clearance.  The 
modes  are  as  follows: 

Standby  Mode-,  the  pilot  has  full  control  of 
the  aircraft  (stick  and  throttle)  and  the 
system  is  initialised  and  ready  for  use. 

-  Free  Track  Mode-,  the  pilot  has  roll  control 
of  the  aircraft  while  the  system  provides 
automatic  height  and  thrust  control.  This 
mode  is  intended  to  support  the  aircrew  in 
deviating  from  pre-planned  routes,  so  that 
threats,  obstacles,  etc.  can  be  safely 
avoided  without  having  to  climb  to  the 
minimum  sector  altitude  (MSA). 

-  Fixed  Track  Mode-,  the  automatic  flight 
along  pre-planned  TF  routes.  The  pilot  has 
the  ability  to  either  manually  fly  the  route 
with  auto-throttle  support  (also  known  as 
the  Manual  Mode)  or  let  the  system  fully 
control  the  aircraft. 

The  flight  guidance  algorithms  consist  of 
several  elements  to  accomplish  the  4D  route 
planning.  Autopilot  and  Flight  Director 
functions. 

-  To  generate  the  4D  routes,  an  algorithm  is 
needed  that  takes  various  factors  into 
account.  Ideally  suited  to  this  application  is 
the  Rolling  Stone  algorithm,  named  so 
because  it  resembles  rolling  a  stone  or  disc 
over  a  terrain  profile  to  generate  the  flyable 
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4D  route  (Figure  5).  Climb/dive  angles, 
push/pull  radii,  set  clearance  height,  speed 
limits,  dynamic  manoeuvring  and  engine 
dynamics  are  some  of  the  more  important 
parameters  that  can  be  adjusted.  The 
Rolling  Stone  algorithm  also  ensures  that 
peaks  and  obstacles  are  over  flown  in  level 
flight  without  vertical  speed  (no 
"ballooning"). 

Energy  Management  (EM)  is  very  important 
to  providing  good  TF  performance  for  a 
transport  aircraft,  as  the  thrust-to-weight 
ratio  is  low.  By  setting  a  speed  range,  an 
obstacle  can  be  over  flown  with  a 
combination  of  static  and  dynamic  climbing 
(Figure  6).  The  aircraft  must  first 
accelerate  to  increase  its  energy  level,  then 
climbs  («2  deg.)  at  constant  velocity, 
before  climbing  dynamically  («10  deg.).  At 
the  peak  of  the  climb,  the  velocity  is  again 
a  minimum  and  a  descent  is  initiated. 

-  Ride  Comfort  enables  the  aircrew  to  set 
the  vertical  acceleration  limits,  or  the 
hardness,  of  4D  route.  This  feature  is 
critical  in  ensuring  a  cargo  is  ready  for 
action  or  use  when  delivered  to  its 
destination.  For  example,  paratroopers 
would  require  a  softer  ride  than  a  payload 
of  tanks.  Two  routes  over  the  Harz 
Mountains  in  Germany  are  shown  in  Figure 
7  for  comparison  of  the  Hard  and  Soft 
Ride. 

•  The  Man/Machine  Interface  consists  primary  of 
inputs  to  the  system  (stick,  pedals,  throttle, 
control  panel)  and  outputs  to  the  aircrew 
(displays,  system  status). 

The  control  panel  enables  to  the  pilot  to 
select  the  mode,  the  TF  route,  the  ride 
mode  and  the  set  clearance  height  (SCH) 
and  shows  the  system  status. 

-  The  displays  used  consist  of  Head  Up 
(Figure  12),  Head  Down  (Figure  13)  and 
Navigation  Displays.  This  combination 
provides  the  pilots  with  all  relevant 
information  and  even  an  enhanced  3D 
synthetic  view  of  the  terrain  in  the  HDD. 
With  increased  situation  awareness,  the 
aircrew's  workload  is  decreased  in 
demanding  situations  (IMC,  night,  etc.). 

•  An  obstacle  cueing/detection  sensor  is  critical 
to  the  verification  of  terrain  database,  as  it 
provides  a  safety  check  against  unmapped  or 
new  obstacles,  which  are  not  known  to  the 
mission  database.  The  use  of  a  millimetre  radar 
or  a  FUR  (forward  looking  infrared)  could  fulfil 
the  requirements  of  this  function. 

•  A  link  to  on-board  mission  planning  station 
provides  the  ability  to  (re)plan  TF  routes  in 
flight,  based  on  new  information  (threat  and 
scenario  updates)  received  by  data  link. 
Additionally,  this  link  serves  to  upload/update 


the  mission  database  before  the  start  of  a 
mission. 


Technology  Demonstration 

Work  in  two  separate  technology  research  areas 
has  enabled  the  low-risk,  partial  realisation  of  the 
concept  described  above  (Figure  8).  Each  research 
area  has  used  common  computer  hardware,  so 
that  the  future  integration  of  individual  projects 
can  be  facilitated. 

The  development  and  flight  demonstration  of 
the  navigation  concept  in  the  RAPIN 
programme. 

The  development  and  demonstration  of  flight 
guidance  and  display  concept  in  the  ATTAS 
and  Airbus  Simulator  to  evaluate  the  handling 
qualities  and  TF  performance  of  the  LLF 
system  [1  ]. 


RAPIN 

RAPIN  (Reliable  Autonomous  Precise  Integrated 
Navigation)  is  a  government-sponsored  research 
and  development  programme  to  investigate  the 
above  mentioned  navigation  concept.  A  more 
detailed  description  of  the  RAPIN  concept  is 
provided  in  [2]. 

The  RAPIN  flight  trials  were  conducted  with  a 
German  Air  Force  Cl  60  Transall  ANA/FRA  from 
the  WTD61.  The  aircraft  test  configuration  (Figure 
9)  consisted  of  a  Collins  GPS  Receiver  (MAGR 
3M),  a  Honeywell  LINS  (H423),  a  Dasa  Radar 
Altimeter  (DRA100)  and  the  aircraft's  Air  Data 
Computer.  A  reference  position  was  obtained 
using  an  Omnistar  DGPS  and,  whenever  possible, 
the  tracking  radar  at  Manching,  Germany.  The 
flights  were  flown  at  low  to  medium  altitude  over 
diverse  terrain  types  (flat  to  rugged,  eg.  Black 
Forest)  and  over  water  (Lake  Constance,  North 
Sea). 

As  the  flight  test  series  was  only  concluded  in 
August  1998,  the  initial  results  are  very  positive 
and  a  sample  is  provided  in  Figure  10.  This  plot 
shows  a  jump  that  occurred  in  the  GPS  position. 
At  this  point  in  time,  the  pilots  noticed  something 
was  wrong  with  the  GPS  and  had  to  disengage  it. 
RAPIN  recognised  this  error  and  subsequently 
ignored  the  GPS  solution,  while  still  providing  a 
very  precise  navigation  solution.  Both  RAPIN  and 
the  reference  system  remained  very  close  for  the 
remainder  of  the  flight,  demonstrating  RAPIN's 
high  reliability  and  precision. 
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FLA  Experiment  Cockpit  Simulator 

The  system  presented  to  and  demonstrated  by 
German  Air  Force  Pilots  in  FLA  Experimental 
Cockpit  Simulator  at  Deutsche  Airbus  in  Hamburg, 
Germany  consisted  of:  mission  database,  full  Care 
Free  Moding,  a  generic  control  panel,  and  4D 
flight  guidance  and  display  functions.  Additionally, 
the  system  was  coupled  to  the  Airbus  Fly-by-Wire 
(FbW)  system  for  full  closed-loop  capabilities. 

The  pilots  were  not  only  required  to  fly  a  standard 
tactical  transport  mission  as  both  pilot  flying  and 
pilot  non-flying,  but  required  to  give  comments  on 
a  LLF  system  in  general  (design  driving  comments) 
and  feedback  on  the  Dasa  system.  Although 
lacking  the  effects  of  flight,  the  TF  performance 
was  very  good  in  both  the  Manual  and  Fixed  Track 
Modes.  A  switch  of  modes  or  tracks  posed  little 
problems  to  the  pilots.  Full  results  including  the 
comments  by  the  pilots  are  available  in  [3]  and  a 
selection  is  presented  below. 

The  major  design  driving  comments  were: 

"...low  level  flight  is  the  only  chance  of 
survival,  penetrating  a  medium/high  threat 
scenario..." 

"...especially  the  freedom  to  perform 
spontaneous  lateral  manoeuvres  with 
terrain  clearance  is  advantageous..." 

"...low  level  flight  with  terrain  clearance 
requires  90%  of  a  pilot's  mental  power;  a 
reduction  of  crew  workload  is  desirable..." 

Some  comments  about  the  system  and  positive 
pilot  feedback  were: 

"...the  successful  flexibility  has  been 
reached,  especially  under  IFR  conditions..." 
"...this  system  is  superior  to  a  Flight 
Management  System..." 

"...the  Free  Track  Mode  reduces  the  crew 
workload  and  is  better  a  pull-up  onto 
minimum  safe  altitude,  and  therefore 
minimises  exposure  to  threats..." 


ATTAS  Flight  Trials 

The  ATTAS  flight  trials  consisted  of  3  flight-test 
campaigns  of  several  flights  each  in  December 
1995,  April  and  September  1997.  The  primary 
goal  was  the  in-flight  demonstration  of  the  display 
concept  (HUD  and  LCD)  and  the  basic  TF  system 
with  German  Air  Force  pilots.  In  each  campaign, 
manual  terrain  following  along  pre-planned  routes 
only  (no  auto-throttle)  was  performed.  This  is 
basically  equivalent  to  the  use  of  the  Standby  and 
Manual  Modes.  The  flight  altitude  over  the  Harz 
Mountains  in  Germany  was  between  9000  and 
1 2000  ft  with  virtual  offset  in  displays  to  simulate 
150ft  SCH. 


Figure  1 1  shows  the  left  side  of  the  ATTAS 
cockpit,  where  the  LCD  was  installed  in  front  of 
the  instruments  and  a  Sextant  HUD  from  above. 
The  FbW  sidestick  and  the  throttles  are  visible  on 
the  lower  left  and  right,  respectively.  The  right 
seat  in  the  ATTAS  is  reserved  for  a  safety  pilot, 
who  can  at  any  time  override  the  FbW  system 
with  the  original  mechanical  controls. 

The  results  indicate  that  low  level  flight  with  this 
system  is  definitely  feasible  and  the  adjustable 
handling  qualities  (ie.  Ride  Comfort)  are  suitable  to 
diverse  mission  requirements  [4].  The  pilot 
followed  the  pre-planned  route  (Figure  14)  and 
stayed  within  the  TF  channel  without  flight  critical 
undershoots  (Figure  15).  The  difference  between 
Standby  and  Manual  Modes  has  been  minimal, 
even  though  one  would  expect  that  the  manual 
mode  with  Flight  Director  would  be  better.  The 
results  show  that  manual  TF  along  the  pre-planned 
routes  requires  an  auto-throttle,  because  pilots 
cannot  perform  EM  manually.  Future  flights  with 
control  inputs  to  the  FbW  (Auto-throttle)  will 
address  this  issue. 


Conclusion  &  Outlook 

The  low  level  flight  guidance  system  being 
developed  by  Dasa's  Military  Aircraft  Division 
fulfils  the  requirements  for  future  tactical  transport 
aircraft,  allows  low  level  flying  without  forward- 
looking  active  sensors  and  can  be  realised  with 
low  technological  risks  based  on  available  COTS 
components. 

The  previous  projects  are  coming  to  a  close  and 
are  currently  being  fused  into  one.  More  ATTAS 
flights  are  planned  for  middle  1999,  where  a 
system  comprised  of  mission  database,  integrated 
navigation,  flight  guidance  and  display  functions 
will  be  flown  and  tested  in  the  Manual  Mode  with 
automatic  thrust  control.  More  long-term  goals  are 
the  full  realisation  of  the  system,  where  the 
obstacle  cueing/detection  and  sensor  fusion  for 
increased  pilot  awareness  is  included,  and  the 
closed-loop  flight  testing  (fully  automatic  TF  in  the 
Fixed  Track  Mode). 
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Abbreviations 


Figures 


ATTAS 

Advanced  Technology  Testing  Aircraft 
System 

DFAD 

Digital  Feature  Analysis  Data 

DTED 

Digital  Terrain  Elevation  Data 

DLR 

Deutsche  Zentrum  fur  Luft-  und 
Raumfahrt 

EM 

Energy  Management 

FLA 

Future  Large  Aircraft 

FUR 

Forward  Looking  Infrared 

FTA 

Future  Transport  Aircraft 

GPS 

Global  Positioning  System 

HDD 

Head  Down  Display 

HUD 

Head  Up  Display 

LATAN 

Low  Altitude  Terrain  Avoidance  and 
Navigation 

LCD 

Liquid  Crystal  Display 

LINS 

Laser  Inertial  Navigation  System 

LFH 

Aviation  Obstacles  (Luftfahrthindernisse) 

LLF 

Low  Level  Flight 

NAV 

Navigation 

ND 

Navigation  Display 

PFD 

Primary  Flight  Display 

RAPIN 

Reliable  Autonomous  Precise  Integrated 
Navigation 

SCH 

Set  Clearance  Height 

TA 

Terrain  Avoidance 

TF 

Terrain  Following 

TRN 

Terrain  Referenced  Navigation 
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Figure  8:  Flight  Demonstration 


Figure  6;  Energy  Management 


Figure  9:  RAPIIM  in  C160  Transall 
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Figure  12:  Head  Up  Display  (HUD) 
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1.  SUMMARY 

A.?  Command  and  Control  systems  are  getting  larger  and 
more  complex,  reliable,  modifiable,  scalable,  platform 
and  operating  system  independent,  distributed  and  non¬ 
proprietary  programming  environment  becomes  an  im¬ 
portant  issue  for  system  engineers  and  developers.  Al¬ 
though  CORBA  (Common  Object  Request  Broker 
Architecture)  concept  and  its  implementations  offer  a 
platform  and  operating  system  independent  software  ar¬ 
chitecture  for  large  scale  systems,  there  are  still  some  im¬ 
portant  missing  issues  such  as  reliability,  fault-tolerancy 
and  real-time/fast  enough  QoS  behaviour.  In  this  paper, 
we  present  our  evaluation  results  based  on  our  experience 
and  efforts  in  designing  a  C2  system  utilizing  currently 
available  COTS  technologies  as  much  as  possible  and 
propose  an  infrastructure  to  satisfy  the  reliability  and 
fault-tolerancy  requirements  of  a  C2  system  software  ar¬ 
chitecture  with  some  additional  domain  interfaces  over 
CORBA.  We  also  present  our  overall  software  architec¬ 
ture  that  will  utilize  this  infrastructure  in  C2  system  do¬ 
main. 

Keywords:  CORBA,  Command  and  Control  Systems, 
Software  Architecture,  Distributed  Programming  Envi¬ 
ronment,  Reliability 

2.  INTRODUCTION 

Military  Command  and  Control(C2)  system  have  to 
be  reliable,  fault-tolerant  and  also  should  meet  the 
real-time  /  fast  QoS  criteria  in  distributed  environ¬ 
ments.  The  feature  “No  single  point  of  failure” 
makes  distributed  systems  more  crucial  for  C2  sys¬ 
tem  architecture.  On  the  other  hand,  very  rapidly 
changing  operating  systems  and  platforms  mandates 
layered  approach  between  the  application  software 
and  underlying  platform  dependent  software  which 
makes  it  easier  to  modify  the  application  programs 
for  new  platforms  with  less  efforts. 

With  the  improvements  in  the  commercial  standards 
and  their  resultant  products,  both  in  terms  of  per¬ 
formance  and  supported  services,  the  usage  of 
COTS  technologies  in  military  systems  becomes 
more  and  more  important  due  to  especially,  the  plat- 


Selahattin  Kuru 

Department  of  Computer  Engineering 
Bogazici  University 
Istanbul,Turkey 
kuru@boun.edu.tr 

form  and  operating  system  independency,  low-cost, 
non-proprietary  and  replacability  features  of  such 
systems. 

The  main  scope  of  this  study  is  to  specify  the 
requirements  of  the  software  architecture  for  C2 
systems,  and  present  our  evaluation  showing  that 
CORBA  architecture  with  some  modifications,  can 
be  used  for  this  domain.  With  a  system,  utilizing  a 
commercial  standards  as  the  underlying  backbone 
architecture,  we  hope  that  system  designers  and 
implementors  will  have  more  flexibility  and  time  to 
deal  with  their  original  problems.  Being  able  to  use 
these  commercial  architectures  will  also  provide 
third  party  utilities  that  will  enable  the  implemen¬ 
tors  deal  with  the  real-life  problems  more  easily. 

As  the  background  information  we  present  the 
requirement  analysis  of  a  C2  system  in  general  and 
some  of  the  features  of  the  existing  distributed 
COTS  architectures  including  a  short  explanation 
on  CORBA  and  its  services,  especially  Event  Serv¬ 
ice,  in  section  3.  We  then  discuss  the  frameworks 
that  we  built  for  the  evaluation  of  distributed  sys¬ 
tems  for  C2  purposes  CORBA  and  the  evaluation 
results  along  with  the  performance  evaluation  of 
CORBA  based  systems  in  section  4.  In  section  5,  we 
present  the  design  and  the  features  of  the  infrastruc¬ 
ture  which  we  believe,  is  especially  useful  in  Event- 
Based  Data-driven  C2  system  domain,  based  on 
CORBA  Event  Service  to  provide  reliability  and 
fault-tolerancy  and  its  usage  in  a  C2  software  archi¬ 
tecture.  Section  6  will  cover  our  conclusions  and 
possible  future  work. 

3.  BACKGROUND 

3.1  Basic  Requirements  for  a  Military  C2  System 

C2  systems  are  the  brain  of  many  military  computer 
systems.  As  an  example  of  a  C2  system,  consider  a 
frigate  with  different  sensors,  weapons  and  commu¬ 
nication  channels  with  different  units  in  the  fleet 
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or  at  the  shore.  A  schematic  diagram  of  such  a 
C2  system  is  given  in  Figure  1 . 


Sensor  1...  nOperator(s) Weapon  l...n 

Command  and  Control  System 


Td 


(To/from)  outside  world 


Figure  1  .Schematic  Diagram  of  a  sample  C2  System 

The  task  of  the  C2  system  can  be  outlined  as 
gathering  the  information  from  different  sensors, 
processing  these  information  with  some  predefined 
or  interactive  criteria,  and  evaluating  the  information 
to  produce  the  necessary  actions  to  be  taken  directly 
by  the  system  or  by  the  command  team. 

Due  to  their  vitality  in  military  systems;  C2  systems, 
have  to  meet  the  real-time  criteria  defined  for  each 
specific  task  and  have  to  be  reliable  in  a  distributed 
manner.  On  the  other  hand  complexity  of  the 
applications  programs  makes  it  desirable  to  have  a 
layered  apporoach  between  these  programs  and  te 
underlying  architecture  in  order  to  have  a  more 
adaptable  programs. 

3.2  Requirements  for  a  Military  C2  System  Soft¬ 
ware  Architecture 

With  the  requirements  specified  in  above  section  and 
our  experience  in  C2  system,  we  set  a  framework 
indicating  the  very  basic  requirements  of  a  C2 
system  software  architecture.  We  think  that  these  are 
the  system  level  requirements  that  can  be  used  in 
order  to  evaluate  a  distributed  architecture  for  our 
purposes. 


Table  1. System  Level  Evaluation  Framework 


Requirement 

Name 

Description 

Stability 

the  degree  which  shows  the  maturi¬ 
ty  of  the  product. 

Simplicity 

the  degree  to  which  the  product  is 
straightforward  to  develop  applica¬ 
tion  programs. 

Hardware  In¬ 
dependence 

The  degree  to  which  the  software  is 
de-coupled  from  the  hardware  on 
which  it  operates. 

Integrable  into 
Hardware 

The  effort  required  to  embedded 
into  different  hardware 

Interoperable 

The  effort  required  to  couple  one 
system  to  another 

Table  1. System  Level  Evaluation  Framework 
_ (con’t) _ _ 


Requirement 

Name 

Description 

Reliable 

The  extent  to  which  a  program  can 
be  expected  to  perform  its  intended 
function  with  required  precision 

Scalability 

The  degree  to  which  a  system  or  ap¬ 
plication  can  handle  increasing  or 
decreasing  demand  on  system  re¬ 
sources  without  significant  per¬ 
formance  degradation.k 

Real-time 

The  degree  to  which  the  software 
monitors/analyses/controls  real- 
world  evens  as  they  occur 

Portability 

The  degree  to  which  a  software 
source  code  remains  unchanged 
even  when  its  environment  chang¬ 
es.  (Ex:  change  of  operating  sys¬ 
tem) 

Support  by  the 
Industry 

The  software  architecture  should  be 
supported  by  the  majority  of  the  in¬ 
dustry. 

Now  that  we  set  a  framework  for  the  general 
evaluation  purposes  of  the  distributed  systems,  we 
can  take  a  look  at  to  the  work  done  in  distributed 
systems. 

3.3  An  overview  for  Distributed  Systems 

As  being  a  very  important  area  of  Computer  Science 
and  Information  Technology,  the  studies  in 
Distributed  Systems  are  very  popular  in  both 
academic  and  commercial  world.  Besides  the 
projects/products  MACH,  AMEOBA,  ACE,  ISIS, 
HORUS,  ELECTRA  developed  in  the  academic 
studies,  there  are  also  very  valuable  efforts  in  the 
commercial  world  such  as  DCOM,  JavaRMI,  DCE, 
and  CORBA.  The  requirements  such  as  stability, 
interoperability  and  portability  leads  us  to  focus  on 
the  commercial  products.  Although  we  covered 
many  academic  study  in  our  research,  they  will  not 
be  considered  here. 

DCOM  is  the  Microsoft’s  solution  for  Windows 
only  platforms,  hence  a  single  vendor  and  single 
platform  solution. 

JavaRMI  (Remote  Method  Invocation)  is  Sun 
Microsystems’  distributed  computing  solution  for 
Java  programming  Language.  Java  is  an  open 
standard  and  runs  almost  on  all  computing 
platforms,  but  it  is  not  mature  and  stable  enough  to 
become  a  C2  middleware  yet. 
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DCE  is  an  open  distributed  computing  standard  and 
implementation  from  OSF  (Open  Software 
Foundation).  It  has  many  similarities  with  CORBA 
but  the  main  drawback  is  its  structured  programming 
paradigm  and  its  lack  of  higher  level  services  such  as 
Naming,  Trader,  Event  or  Security  services  of 
CORBA. 

CORBA  as  explained  in  [1]  is  the  core  of  Object 
Management  Architecture  (OMA)  introduced  by 
Object  Management  Group  (OMG).  The  biggest 
ever  software  consortium  with  800  participating 
companies.  CORBA  is  also  capable  for  very  large 
scale  distributed  computing  such  as  in  the  Internet. 

3.3.1CORBA  and  Common  Object  Serv¬ 
ices. 

Due  to  the  ever  increasing  need  for  reusable  compo¬ 
nents  that  have  well  defined  interfaces,  Object  Man¬ 
agement  Group  (OMG)  introduced  an  Object 
Management  Architecture(OMA)  to  solve  some  of 
the  problems.  OMA  has  four  components.  These  are 
Object  Request  Broker(ORB),  Object  Services  (OS) 
,  Common  Facilities  (CF)  and  Application  Objects 
(AO)  (Figure  2).  ORB,  a  kind  of  software  bus,  is 

Application  Objects  Common  Facilities 

??  ??? 

(  ORB  ) 

tttT 

Object  Services 


Figure  2.  The  Components  of  Object  Management 
Architecture  (OMA) 

the  core  of  this  architecture.  CORBA  is  the 
arhitectural  specification  of  ORB  component.  All 
object  interfaces,  including  common  facilities, 
object  services  and  ORB  itself  are  defined  with  in  an 
Interface  Definition  Language  (IDF).  There  are 
various  mappings  from  IDL  to  other  programming 
languages  such  as  C,  C-n-,  lava,  Ada  and  COBOL. 
IDL  is  used  to  define  the  interface  of  CORBA 
objects.  It  is  the  main  tool  to  describe  distributed 
object  oriented  architecture. 

Besides  basic  functionalities  that  ORB  provides, 
many  services  that  are  found  to  be  useful  by  OMG 
are  introduced  into  OMA  as  Object  Services.  Basic 
Object  Services  can  be  listed  as:  Naming,  Event, 
Security,  Transactions,  Trading,  Life  Cycle, 
Externalization,  Licensing,  Time,  Property, 
Relationships,  Concurrency  Control,  Persistent 
Objects,  Query  services. 


Out  of  these  services  especially  Event  Service 
provides  a  good  mechanism  for  large-scale 
distributed  programs  including  C2  systems. 

Event  Service  presents  a  view  of  an  event-based 
messaging  environment.  Suppliers  generate  events 
and  consumers  receive  them.  It  models  a  decoupled 
communication  style. 

The  Event  Service  supports  asynchronous  message 
delivery  and  allows  one  or  more  suppliers  to  send 
messages  to  one  or  more  consumers.  Event  data  can 
be  delivered  from  suppliers  to  consumers  without 
requiring  these  participants  to  know  about  each  other 
explicitly [2].  Event  service  design  is  scalable  and  is 
suitable  for  distributed  environments.  There  is  no 
requirement  for  a  centralized  server  or  dependency 
on  any  global  service[3]. 

4.  EVALUATION  OF  DISTRIBUTED 
SYSTEMS  FOR  C2  PURPOSES 

In  the  evaluation  phase,  we  utilized  the  requirements 
that  we  specified  in  the  last  section  and  our 
experience  on  C2  so  far.  Although  performance 
evaluation  has  also  been  considered,  more  effort  has 
been  put  into  the  architectural  evaluation. 

4.1  Architectural  Evaluation 

For  the  architectural  evaluation  of  different 
distributed  systems  for  C2  purposes,  we  used  the 
table  set  in  the  section  3.2  and  the  results  are  given 
in  Table  2. 


Table  2.System  Level  Evaluation  Results 


DCOM 

RMI 

DCE 

COR 

BA 

stability 

yes 

yes 

yes 

yes 

simplicity  to 
build  large  scale 
systems 

yes 

yes 

no 

yes 

hardware  inde¬ 
pendence 

no 

yes 

most 

yes 

integrable  into 
hardware 

yes 

yes 

no 

yes 

interoperable 
with  other  m/w 

no 

yes 

yes 

yes 

reliable 

no 

no 

no 

no 

real-time 

no 

no 

no 

no 

portability 

no 

yes 

yes 

yes 
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As  a  result  of  evaluation  of  the  first  framework,  it 
turned  out  to  be  that  among  DCOM,  RMI,  DCE  and 
CORBA,  with  some  nice  services  and  the  support  of 
the  industry,  CORBA  satisfies  more  requirements 
than  the  others. 


Table  3.  Framework  for  usability  of  CORBA 
products  in  C2  systems 


Requireme 
nt  Name 

Description 

Visi 

brok 

er 

Orb 

ix 

Fault  Tol¬ 
erant  ORB 

Distributed  ORB  archi¬ 
tecture  prevents  single 
point  of  failure. 

Yes 

No 

Fault  Tol¬ 
erant  Ap¬ 
plication 
programs 

A  system  wide  fault-toler¬ 
ant  service  should  follow 
the  applications’  state  and 
if  there  is  any  crashed  ap¬ 
plication,  new  applica¬ 
tions  are  to  be  created 
automatically  by  restoring 
the  old  state  of  the  appli¬ 
cations. 

No 

No 

Persistent 
Data  Serv¬ 
ice  (PDS) 

There  are  some  data  to  be 
kept  in  the  system  forever. 
These  data  should  be 
saved  in  the  files. 

No 

Yes 

Fault  Tol- 
erancy  in 
PDS 

These  permanent  data  are 
to  be  kept  in  more  than 
one  node  or  in  every  node. 

No 

No 

Critical 

Data  Serv¬ 
ice  (CDS) 

There  are  some  data  to  be 
kept  in  the  system  while 
the  system  is  running. 

This  kind  of  data  is  to  be 
cached  in  each  of  the 
nodes  for  reliability  pur¬ 
poses. 

No 

No 

Fault  Tol- 
erancy  in 
CDS 

These  cashed  data  are  to 
be  kept  in  more  than  one 
node  or  in  every  node. 

No 

No 

Event 
Trigger¬ 
ing  Filter 
mecha¬ 
nism. 

Listener  objects  could 
specify  on  which  values 
of  the  event  information 
fields  they  are  interested 
in.  These  filters  might  be 
the  source  of  the  event  or 
any  field  of  the  informa¬ 
tion  sent  by  the  event. 

No, 

* 

Yes 

*  but  interceptors  for  callbacks  can  be  used 


Table  3.  Framework  for  usability  of  CORBA 
products  in  C2  systems  (con’t) 


Requireme 
nt  Name 

Description 

Visi 

brok 

er 

Orb 

ix 

Event  Log 
key  field 
indica¬ 
tions. 

Each  listener  object  could 
specify  the  “key”  fields 
that  it  is  interested  in,  in 
order  to  avoid  overwriting 
valuable  info  or  stocking 
unnecessary  information. 

No 

No 

Event  trig¬ 
gering  for 
more  than 

one  event 
(for  an 
event  list) 

Each  listener  could  speci¬ 
fy  a  list  of  events  that  it  is 
to  be  invoked  in  any  one 
of  these  events  occurs. 

No 

No 

Event 
waiting  for 
a  specific 
time  peri¬ 
od. 

Each  listener  could  speci¬ 
fy  a  time-out  value  to  wait 
for  an  event. 

No 

No 

Concept  of 
new  and 
old  data  in 
event  in¬ 
formation 

When  an  event  is  to  be 
read  by  the  listener,  it  be¬ 
comes  an  old  data  for  it, 
but  it  may  be  still  valuable 
in  case  of  crashes,  or  for 
some  statistical  purposes. 

No 

No 

Time  syn¬ 
chroniza¬ 
tion 

All  of  the  nodes  in  the  dis¬ 
tributed  environment  ' 
could  use  the  same  time 
values  in  order  to  be  pre¬ 
dictable  in  their  behav¬ 
iours 

No 

No 

Distribut¬ 
ed  File 

manage¬ 

ment. 

For  fault-tolerancy  pur¬ 
poses,  same  file  structure 
is  to  be  kept  in  different 
nodes  of  the  distributed 
environment.  Hence  a  file 
manager  which  organizes 
mirror  files  for  the  re¬ 
quired  information  is  a 
need. 

No 

No 

For  further  evaluation,  CORBA  based  products 
were  considered  for  their  suitability  against  the 
requirements  of  a  heterogeneous  software  architec¬ 
ture  [4],  combining  event  based,  implicit  invoca¬ 
tion,  publish-subscribe,  repository  and  object 
oriented  styles[4],  in  an  harmony.  An  evaluation 
framework  and  evaluation  results  for  two  sample  im¬ 
plementation  is  presented  in  Table  3. 


23-5 


As  the  result  of  the  architectural  evaluation,  we  see 
that  although  CORBA  provides  a  good  environment 
from  many  perspective,  there  are  still  some  missing 
issues  in  the  specifications  especially  for  the  real¬ 
time,  reliable  features  of  C2  systems.  Time  synchro¬ 
nization,  system  monitoring  and  general  file  man¬ 
agement  issues  are  also  problematic  areas  which 
will  effect  the  design  of  a  such  architecture. 

4.2  Performance  Evaluation: 

Another  aspect  of  the  evaluation  is  of  course  the 
performance  issue.  One  thing  that  is  to  be  considered 
here  is  that  the  performance  of  CORBA  applications 
may  differ  greatly  depending  on  the  usage  of  the 
very  little  details  of  CORBA  implementations. 
Hence  those  figures  might  be  misleading.  On  the 
other  hand,  increasing  processing  power  of  the 
platforms  and  fine-tuning  of  the  CORBA 
implementations  by  the  companies  provide  better 
performance  figures  every  day. 

Hence  rather  than  specifying  some  of  the  numerical 
values  in  tables,  (although  we  have  many  test  cases), 
we  prefer  to  give  the  results  out  of  these  performance 
tests. 

We  have  done  some  tests  in  two  different  test 
environments  resulting  10  times  better  performance. 
The  first  test  environment  was  consisting  of  a  Sparc- 
Ultral  and  a  Sparc  Notebook  with  lOB  Ethernet.  The 
second  environment  that  we  set  up  very  recently 
consists  of  4  Ultra-60  machines  with  155MB  ATM 
connections.  This  reveals  the  effect  of  the  hardware 
on  the  performance.  Another  effect  on  the 
performance  is  to  use  the  appropriate  methods 
according  to  the  selected  ORB.  This  is  actually  the 
fine-tuning  part  of  the  ORB.  Some  implementations 
of  ORB  may  perform  better  for  some  given 
scenarios. 

During  these  tests,  we  realized  that  messaging  load 
is  one  of  the  key  aspects  to  consider  for  the 
performance,  hence  it  must  be  carefully  monitored. 
Messaging  can  be  decreased  by 

1. Grouping  objects  into  a  single  server  process, 

2. Using  in-process  event  channels, 

3. Keeping  request  and/or  event  push  frequencies  at 
medium  levels  (~10Hz), 

4. Retuming  large  data  structures  at  once  instead  of 
having  to  make  multiple  requests  for  clients  on  the 
same  host 

5. Reducing  the  number  of  clients  accessing  the  event 
channel  in  the  same  host. 

These  methods  will  help  the  designed  system  be 


predictable  without  pushing  the  limits  of  underlying 
hardware  and  operating  system. 

4.3  Evaluation  Summary 

As  the  result  of  the  continuing  experience  and  eval¬ 
uation,  we  see  that  in  a  CORBA  based  software 
architecture  for  C2  systems,  there  are  two  risks 
areas.  (Real-time)  performance  and  reliability  fea¬ 
tures.  Time  synchronization,  system  monitoring  and 
general  file  management  issues  are  also  problematic 
areas  which  will  effect  the  design  of  a  such  architec¬ 
ture. 

Out  of  these  evaluations,  we  concluded  that 
CORBA  architecture  as  currently  specified  does  not 
support  enough  features  necessary  to  build  reliable, 
large  scale  C2  systems.  Since  CORBA  does  not  pro¬ 
vide  a  completely  appropriate  software  architecture 
for  C2  systems,  what  we  decide  is  to  get  out  of 
CORBA  as  much  as  we  can  get  and  build  an  inter¬ 
mediate  layer  on  top  of  this  architecture  to  fulfil  our 
requirements  with  a  minimum  size  of  in-house 
implementation  [5]. 

4.4  Related  Research 

There  are  some  on-going  research  to  increase  the 
capabilities  of  OMA  specifications.  Adding 
reliability  to  CORBA  architectures  is  also  a  point  of 
interest  for  many  research  group.  Orbix-f-Isis  is 
offered  to  provide  a  reliable  environment  for  Event 
Service  of  CORBA[6].  Server  objects  are  to  be  used 
via  Object  Groups  and  a  log  file  for  the  event  history 
is  to  be  kept  either  in  the  memory  or  in  the  disk.  For 
such  an  architecture  “Event  Log  key  field 
indication”  and  “Fault  Tolerancy  in  Persistent  Data 
Service”  can  be  listed  as  the  missing  requirements 
with  respect  to  Table  1.  There  is  also  no  indication 
for  the  “fault  tolerant  ORB”  in  their  studies.  In  case 
of  a  new  receiver  arrives  to  the  system,  (e.g.  due  to 
the  crash  of  the  old  receiver),  they  indicate  that  a 
user-defined  number  of  events  will  be  backed  up 
from  the  log  file.  In  our  case  this  approach  is  not 
appropriate,  the  number  of  the  event  information  to 
be  backed  up  is  not  predictable  at  all. 

Another  product  CyberBus[7],  based  on  source-code 
free  partial  CORBA  implementation  ELECTRA  [8], 
has  been  announced.  It  has  several  differences  from 
the  CORBA  Event  Service.  Main  subject  for  our 
case  is  that  it  introduces  active  and  passive  replicas 
of  the  objects  for  fault  tolerancy.  A  mechanism  will 
determine  the  crashed  objects  and  notifies  surviving 
objects  of  such  events.  Additionally,  CyberBus  can 
be  configured  to  automatically  restart  a  crashed 
object  on  a  new  machine,  without  having  to  restart 
the  objects  that  interacted  with  the  crashed  object.  In 
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this  study,  also  virtual  synchrony  concept  has  been 
utilized. 

The  research  group  at  Washington  University 
evaluated  CORBA  products  for  the  reliability, 
flexibility,  reusability  and  performance  features. 
Very  well  defined  performance  tests  have  been 
applied  for  the  commercial  CORBA  products  and 
also  some  solutions  for  the  possible  bottleneck 
points  have  been  suggested  by  this  group.  They  offer 
a  system  which  satisfies  real-time  ORB[14]  that  can 
deliver  end-to-end  QoS  guarantees  to  applications. 
In  their  work  they  propose  some  extensions  to 
CORBA  Event  service  so  that  it  will  be  suitable  for 
real-time  systems.  These  extensions  support  periodic 
rate-based  event  processing  and  efficient  event 
filtering  and  correlation  [2]. 

The  inefficiency  of  CORBA  for  reliable  and  high 
available  systems  are  explained  in  detail  in  [9].  Also 
in  this  paper,  the  interesting  methods  for  building 
reliable  systems  namely,  Message  Queues, 
Transaction  Processing  Monitors  and  Virtual 
Synchrony  methods  are  explained  and  evaluated. 
Orbix-fisis  is  offered  to  provide  a  reliable 
environment  for  Event  Service  of  CORBA[6]. 
Another  middleware  CyberBus,  which  runs  on 
source  code  free  partial  CORBA  implementation 
ELECTRA[8],  has  been  proposed  with  similar 
facilities  to  Event  Service.  Experience  over  the  past 
several  years  illustrates  that  CORBA  is  well-suited 
for  best-effort,  client-server  applications  running 
over  conventional  local  area  networks.  However, 
building  highly  available  applications  with  CORBA 
is  much  harder[9]. 

Although  there  are  some  definitions  and  evaluations 
about  tbe  methods  for  building  reliable  systems  on 
CORBA  architectures,  in  most  of  the  studies,  the 
concept  of  group  communication  and  virtual 
synchrony  have  been  utilized.  We  believe  that  for 
our  case,  in  an  environment  where  the  data  flows 
between  the  processes  are  encapsulated  as  objects; 
utilizing  object  group  concept  for  every  object,  may 
be  a  performance  bottleneck.  On  the  other  hand 
methods  such  as  message  queues  or  TP  has  got  little 
attention.  We  think  that  message  queues  and  data 
redundancy  can  also  be  effectively  used  to  build 
reliable  systems. 

On  the  other  hand,  OMG  has  published  a  Request  for 
Proposal  (REP)  for  the  fault  tolerant  CORBA  using 
entity  redundancy[16]  on  mid  April  98,  which  has  a 
submission  due  date  as  October,20, 98.  We  hope  that 
this  will  increase  the  motivation  for  the 
implementation  of  a  fault-tolerant,  reliable  CORBA 


architecture  in  the  industry. 

5.  PROPOSED  ARCHITECTURE  FOR  C2 
SYSTEMS 

Before  explaining  the  architecture  itself,  we  will 
present  some  information  about  tbe  underliying 
communication  mechanism. 

5.1  Event-based  or  Client-server  architecture 

CORBA  IDL  operation  calls  are  an  ideal  mechanism 
for  communicating  between  a  client  and  a  server:  an 
object  in  a  server  can  advertise  its  IDL  interface  and 
clients  can  use  this  interface  using  simple  and 
familiar  programming  constructs.  A  C++  function 
call  can  result  in  the  activation  of  an  object  on  a 
different  node,  running  on  a  different  operating 
system,  and  writtenn  in  a  different  programming 
language. 

However,  this  direct  point-to-point  communication 
is  not  ideal  when  a  client  must  make  the  same 
operational  call  on  a  set  of  objects.  Having  to  make 
a  sequence  of  operation  calls  is  very  expensive  for 
the  program,  and  inefficient  to  execute.  Such 
communication  is  also  inappropriate  if  a  client  needs 
to  make  a  variable  set  of  objects  that  the  client  can 
not,  or  should  not,  be  able  to  enumerate.  A  client 
may  not  know  the  objects  that  it  needs  to 
communicate  with  because  this  set  is  very  variable 
(so  it  would  be  diffucult  to  maintain  a  list  of  object 
references  in  the  client)  or  because  the  client  and  the 
target  objects  should  not  know  of  each  other. 

On  the  other  hand,  there  are  some  application  level 
programs  and  functionalities  that  they  are  used  by 
many  clients  in  the  system.  Track  Manager  can  be 
given  as  the  best  example  for  such  services.  This 
kind  of  services  are  accessed  by  many  clients,  and 
they  have  well-known  interfaces.  For  such  purposes, 
client-server  architecture  should  be  the  choice. 
Hence  implementing  such  a  functionality  as  a  server 
object  will  reduce  the  amount  of  processing  in  the 
other  processes.  Time  server.  File  Management 
Server  can  also  be  considered  as  such  services. 

Even  we  design  to  use  client-server  architecture  for 
some  specific  purposes,  the  implementation  of  the 
server  on  the  other  end,  will  also  be  using  Event- 
service  mechanism  to  keep  its  information  upto 
date. 

As  a  result,  we  can  say  that  Event-Service  will 
be  the  main  communication  facility  in  the  system, 
but  for  specific  purposes  Client-Server  architectures 
will  be  utilized. 
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5.2  Overview  of  the  Architecture 

We  present  a  domain  specific  CORBA  based 
architecture  in  this  section.  Our  overall  design  for  a 
C2  system  architecture  is  shown  in  Figure  3. 

In  the  “infrastructure”  layer,  we  would  like  to  handle 
all  of  the  communication  and  data  transfer,  data 
adaptation  and  reliability  problems,  where  in  “core 
services”  we  plan  to  perform  system  management 
functionalities.  “Core  applications”  are  the  basic 
applications  that  are  to  be  utilized  by  the  application 
programs  very  heavily  in  this  domain,  and 
“application  programs”  are  the  tactical  applications 
that  will  be  changing  mostly  according  to  the 
platforms  and  the  purpose  that  the  system  is  used  for. 
Out  of  these  four  layers  we  think  that  first  two  can  be 
used  for  many  C2  systems  with  different  purposes, 
where  as  for  “core  applieations”  some  modifications 
or  replacements  might  be  necessary,  and 
“application  programs”  are  very  specific  for  each  C2 


Figure  3.  Overall  Design  for  a  C2  system 

As  it  can  be  seen  from  Figure  3,  the  infrastructure 
has  a  special  importance  in  the  overall  architecture. 
Hence,  our  main  concern  will  be  the  details  of  this 
layer  handling  the  underlying  communication  and 
reliability  features  in  a  data  and  event  driven  sys¬ 
tem. 

5.3  Important  features  of  the  architecture. 

The  layers  of  “Infrastructure”  and  “Core  services” 
have  special  importance  since  these  two  layers 
provide  the  necessary  services  for  implementing  a 
C2  system  regardless  of  its  functionalitiy  and  its  size. 

Tactical  Applications  uses  the  services  provided  by 
the  underlying  layers  and  essentially  gets  the  data, 
performs  some  actions  on  the  data  and  output  the 
result  into  the  system  or  to  the  user.  In  some  cases, 
they  act  as  a  client  of  the  “core  applications”. 

Core  applications  are  most  likely  to  be  implemented 
as  a  client-server  manner  due  to  their  well-known 


interface  for  the  application  programs.  But,  on  the 
lower  level  of  their  interface,  they  will  still  be  using 
subscribe-publish  mechanism  in  order  to  retrieve 
information  from  the  system. 

“Core  services”  layer  is  the  layer  where  functionality 
independent  behaviour  is  embedded,  hence  we 
expect  a  set  of  general  “core  services”  for  each  of  the 
C2  system.  In  our  definition  a  subsystem  is  a  source 
of  or  the  user  of  an  information  in  the  domain. On  the 
other  hand,  domain,  is  a  collection  of  subsystems, 
and  a  system  is  a  collection  of  domains.  Hence  a 
unique  “Domain  Administrator”  for  the  whole 
system  is  required  where  as  for  each  domain  a 
“Domain  Manager”  is  used  to  keep  the  associated 
subsystem  records  in  the  domain. 

As  an  example  for  domain  concept,  platform 
domain,  OTC  domain,  fleet  domain  can  be  given.  In 
each  of  these  domains,  there  will  be  different 
subsystem  assigned  to  this  domain. 

On  the  other  hand,  for  the  “Infrastructure”  layer,  data 
is  the  key  element  in  the  system.  The  Application 
programs  (tactical  /  core  applications  and  core 
services)  simply  specify  their  interest  with  specific 
pieces  of  data,  and  an  event  (or  notification)  service 
with  the  help  of  an  agent  collects  (cashes)  the  data 
from  the  overall  system  for  the  application 
program(s). 

On  the  cashed  data,  the  state  information  of  the 
application  programs  is  maintained,  hence  in  case  of 
crash  of  an  application,  the  new  application  can 
restore  its  state  by  utilizing  this  cashed  data  in  the 
same  or  different  node  of  the  system. 

Due  to  the  special  characteristics  of  the  C2  domain, 
data  instances  represent  mostly  continuous 
quantities  [10],  which  allows  a  slight  relaxation  of 
the  consistency  requirements  normally  associated 
with  distributed  systems.  For  this  kind  of 
information,  there  is  no  need  for  rigid 
synchronization  of  the  cashed  data. 

For  the  data,  that  does  not  fit  into  the  above  category, 
(i.e.  not  continuous)  the  delivery  time  is  a  major 
issue.  For  this  purpose,  effective  notification 
mechanism  is  to  be  provided. 

In  tbe  cashed  data,  data  is  stored  according  to  the  key 
values  specified  by  the  producers.  In  addition  to  this 
storage,  each  application  can  also  specify  different 
key  values  for  other  storage  structures. 

On  the  other  hand,  application  programs  can  specify 
filters  for  the  data  that  they  are  interested  in.  This 
mechanism  is  also  used  to  optimize  the  network 
message  traffic. 
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Query  mechanism  can  also  be  set  on  to  the  cashed 
data  by  the  application  program.  Such  a  mechanism 
is  especially  useful  for  the  effective  triggering  of  the 
application  programs. 

Correlation  mechanism  [2]  which  gives  the 
opportunity  to  the  application  program  to  specify  its 
interest  for  more  than  one  events  is  also  a  desirable 
feature  of  the  system. 

On  top  of  such  an  architecture,  a  system  monitoring 
service  is  also  needed  to  detect  the  possible  crashed 
application  programs  and  restart  them  in  the 
appropriate  places. 

CORBA  and  its  event  service  fits  quite  well  for  the 
underlying  communication  mechanism.  The  reason 
that  we  say  “quite”  have  been  explained  in  section  2, 
as  the  missing  important  features  of  CORBA  system. 
On  top  of  this  basic  layer,  we  are  more  concerned  on: 

l.A  layer  that  will  satisfy  the  additional 
requirements  of  missing  features, 
l.A  layer  that  will  hide  the  CORBA  interfaces  from 
the  application  software  and  provide  an  interface  for 
the  additional  requirements,  and 
3.Design  of  Business  Object  Model  in  the  overall 
architecture.  Namely,  Process  Objects  and  Entity 
objects  are  to  be  defined  and  designed  in  the  system. 

In  a  system  utilizing  publish-subscribe  architectures 
data  plays  the  key  role.  Data  as  being  the  informa¬ 
tion,  passing  between  the  “Process  Objects”  of  the 
Business  Object  Model,  will  be  encapsulated  as 
“Entity  Objects” 

Three  different  types  of  “entity  objects”  have  spe¬ 
cial  meanings  in  the  system.  “Permanent”,  “Criti¬ 
cal”  and  “Periodic”  entities.  These  entities  will  be 
made  accessible  by  using  the  IDL  interface  of  the 
system. 

Permanent  entity  objects  are  to  be  stored  on  each 
node’s  disk  in  the  network,  so  that  application  soft¬ 
ware  can  use  this  information  for  the  initial  start  up. 
(Some  default  settings,  vital  information  etc.) 

Critical  entity  objects  are  to  be  stored  in  the  main 
memory  of  each  node  (heap  /  shared  memory).  To 
avoid  unnecessary  pile-up  of  objects,  objects  with 
the  same  values  of  indicated  “key”  fields  will  be 
overridden  by  the  data  produced  later  (key  fields 
will  be  designated  by  the  producers  of  the 
data).Addionally,  each  consumer  as  being  a 
processing  object  should  be  able  to  indicate  how  the 
entity  object  is  to  be  stored  for  their  application  pur¬ 
poses,  hence  rather  than  a  unique  producer  key  list, 
each  consumer  is  to  prepare  a  key  list  of  that  object 


data  to  be  stored  in  the  memory.  Our  point  is  that  in 
order  to  support  the  reliability,  on  each  node  on  the 
system  network,  the  same  data  is  to  be  kept  as  “crit¬ 
ical”.  In  case  of  a  node  failure,  this  critical  info  will 
be  retrieved  from  one  of  the  other  (master)  nodes. 


Periodic  entity  objects  are  the  objects  for  which  no 
storage  is  required.  Such  objects  constitutes  an 
important  amount  of  data  passing  around  in  a  C2 
system,  and  due  to  the  continuous  nature  of  such 
data,  the  requirement  of  having  all  of  the  data  and 
synchronization  of  such  data  in  different  nodes  is 
not  strictly  necessary.  The  current  event  service 
implementations  fulfil  this  requirement. 

With  a  good  abstraction  we  expect  to  have  1500- 
2000  different  entity  objects  in  a  large  scale  C2  sys¬ 
tem.  Out  of  1500,  approximately  400  periodic,  200 
permanent  and  900  critical  entities  are  expected. 
During  the  design  phase,  this  property  of  the  entity 
object  will  be  set  in  the  IDL  interface  as  an  attribute 
“type”.  Hence  it  will  be  available  through  out  the 
system. 


The  design  of  event  channel  usage  is  also  an  impor¬ 
tant  concept  to  consider.  The  amount  of  event  chan¬ 
nels  and  the  amount  of  different  entity  objects 
passed  through  these  event  channels,  is  one  of  the 
major  design  problem. 


Keeping  these  ideas  in  mind,  we  present  the  general 
schema  of  the  overall  architecture  in  Figure  4. 
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Figure  4. General  Schema  of  the  Proposed 
Architecture 


In  Figure  4,  we  illustrate  all  of  the  major  components 
of  infrastructure  layer  with  respect  to  the  general 
schema  of  the  overall  architecture.  Hence,  this  figure 
does  not  reveal  the  complete  architecture,  but  gives 
an  idea  about  where  the  infrastructure  lies  in  the 
overall  architecture.  All  of  the  components  except 
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the  “Process  Object  Layer”  elements  are  the 
elements  of  the  “infrastructure”  or  “core  services” 
layer. 

Especially  System  Monitoring  service  has 
significant  importance  for  the  reliability  of  the 
system.  That  service  will  detect  the  crashed  “entity” 
or  “process”  objects  and  re-start  or  activate  another 
copy  of  the  crashed  object.  Since  the  “critical” 
entities  are  available  through  out  the  system,  the 
object  will  restore  its  state  prior  to  the  crash. 

Now,  there  are  two  questions  to  be  answered  for 
such  a  nice  mechanism. 

1 .  What  is  the  fail  detection  mechanism? 

2.  What  is  the  mechanism  for  the  synchronization  of 
the  “critical”  data  in  each  of  the  node? 

As  we  will  be  explaining  in  later  in  this  section,  the 
answer  for  these  questions  is  the  notification  service. 
For  the  current  prototype  purposes,  we  consider  that 
each  object  will  report  itself  to  the  subsystem 
manager  by  sending  a  heart  beat.  Subsystem 
manager  will  evaluate  these  information  acc.  to  a 
dynamic  time  criteria  set  w.r.t  the  cpu  usage  and  may 
declare  the  crash  of  an  object.  On  the  other  hand,  we 
postponed  the  synchronization  problem  until  a  good 
notification  service  becomes  available. 

Entity  Object  Manager  Layer  (ML)  and  I/O  Layer 
(lOL)  in  Figure  4  support  the  idea  of  layered 
approach  and  provides  necessary  caching 
mechanism  and  application  interfaces  for  reliability 
and  fault-tolerancy  that  are  missing  in  standard 
CORBA  specs  for  C2  system  domain. 

The  main  idea  is  to  store  the  “critical”  entity  objects 
required  by  the  consumer  applications  (process 
objects)  in  a  redundant  manner,  so  that  a  crushed 
objects  can  recover  to  its  original  state  prior  to  crush 
by  utilizing  these  data  in  the  same  node  or  in  another 
node.  For  “persistent”  entity  objects,  the  storage  is  to 
be  made  to  the  disk  for  initial,  default  set  up  values. 
For  “periodic”  entity  objects  basically  there  is  no 
storage  requirement  and  CORBA  Event  Service  fits 
quite  well  for  this  purpose. 

In  genereal  each  entity  manager  for  the  consumer 
application  programs  will  collect  the  requests  for 
each  of  the  entity,  from  the  application  programs, 
establish  the  event  channel  connections  accordingly, 
and  notify  the  consumer  application  programs,  if  the 
application  program  requests  also  a  notification  for  a 
specific  entity.  It  will  also  cache  the  incoming 
entities  according  their  specific  natures. 

Besides  these  common  services  of  the  managers, 
there  are  two  more  services  that  we  evaluate  as  nec¬ 


essary  for  “critical”  and  “periodic”  entity  object 
managers.  These  are  query  and  correlation  function¬ 
alities.  Application  programs  should  be  able  to 
specify  their  queries  for  the  related  entities. With 
correlation  mechanism  [2],  the  opportunity  to  spec¬ 
ify  its  interest  for  more  than  one  events  (OR  opera¬ 
tion)  will  be  provided  to  the  application  program. 
Other  than  these,  filtering  mechanism  is  another 
feature  that  we  consider  as  necessary  but  we  know 
that  notification  service  will  take  care  of  this  func¬ 
tionality.  Still  entity  object  managers  will  have  to 
provide  an  interface  for  this  feature. 

The  interaction  of  Critical  Entity  Manager(CEM) 
with  other  units  is  given  as  an  example  in  Figure  5. 


Figure  5. Detailed  representation  of  the  CEM  part  of 
the  proposed  infrastructure 


On  every  node,  there  will  be  a  CEM  server,  one  of 
them  acting  as  a  Master.  In  case  of  new  node  com¬ 
ing  up,  this  master  will  supply  the  current  critical 
entity  object  info  to  the  new  CDM  with  only  pro¬ 
ducers’  key  values. 

Currently,  we  are  working  on  a  prototype  that  has 
consumers  and  producers  which  are  not  object  ori¬ 
ented  application  programs.  It  is  due  to  the  risk 
management  concept  arising  from  a  quite  amount  of 
people  which  are  happy  with  their  programming 
API  and  environment  and  design  logic.  That  is  why, 
at  the  first  stage,  a  conversion  from  entity  object  to 
object  data  is  essential  at  least  when  presenting  the 
entities  to  the  application  programmer.  This  kind  of 
adapter  functionality  has  to  be  taken  care  of  by  the 
entity  manager  too.  As  noticed  in  Figure  5,  we  are 
using  shared  memory  and  message  queue  for  the 
interprocess  communication  between  the  applica¬ 
tion  programs  and  the  CEM.  effective  data  structure 
utilization  is  also  a  must  for  performance  reasons. 

On  the  other  hand,  lOL-lib  (Entity  Object  Input/ 
Output  Layer)  provides  a  mechanism  to  hide  the 
details  of  accessing  the  Entity  Object  Data.  This 
layer  is  implemented  as  a  dynamic  library  and  han¬ 
dles  the  communication  between  the  CEM  and  the 
application  layer. 
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For  the  producer  application  programs,  each  of  the 
programs  will  have  a  interface  hiding  the  details  of 
the  CORBA  and  Event  Service,  provided  by  the 
lOL-lib,  but  via  lOL-lib  will  utilize  the  Event  Serv¬ 
ice  or  ORB  directly,  without  any  services  of  Man¬ 
ager  Layer.  In  order  to  decrease  the  network  load, 
we  might  consider  to  add  another  functionality  into 
the  manager  layer  to  take  care  of  the  buffering 
issues  for  publishing  big  chunks  of  data,  rather  than 
many  small  data  into  the  Event  Service. 

Although  being  able  to  cache  the  redundant 
“critical”  entity  objects  is  a  big  step,  we  still  need 
two  more,  very  important  services  to  provide  a 
reliable  infrastructure.  These  are  system  monitoring 
and  management  services.  An  example  of  such  a 
system  has  been  presented  in  [9]. 

5.4  Effects  of  New  COS  Services  for  the  Design 
of  Infrastructure 

Since  OMA  and  its  services  are  rapidly  evolving,  we 
need  to  consider  the  new  possible  services  and  their 
effects  for  the  design  of  the  infrastructure.  As  it  is 
pointed  out  through  the  explanation  of  the 
infrastructure,  there  are  some  problematic  areas, 
which  we  think  new  service  specifications  will  be 
useful. 

One  of  the  main  services  that  we  think  it  will  be 
useful  is  Objects-by-value.  This  service  will  provide 
better  efficiency  for  the  underlying  communication 
mechanism  and  will  let  us  to  encapsulate  the  data 
structures  as  objects.  Currently  we  are  using  data 
structures  to  pass  this  data  among  the  process 
objects. 

Another  important  service  is  the  notification  service. 
With  this  service  QoS,  Assured  Notification, 
Prioritization  and  Filtering  features  will  be  added  to 
the  system.  We  think  this  service  will  be  useful  for 
the  detection  of  faulty  application  programs  and  the 
synchronization  of  the  cashed  data  with  its  assured 
notification  mechanism.  If  the  performance  of  the 
notification  service  becomes  a  problem,  we  might 
use  event  service  and  notification  service 
concurently  serving  different  entity  objects,  namely 
“periodic”  and  “critical”  respectively. 

We  do  not  want  to  go  through  for  the  real-time 
service  of  OMA.  It  will  definitely  effect  the 
deterministic  behaviour  and  hopefully  the 
performance  of  the  system. 

For  Fault-tolerant  service  by  using  redundant  entity, 
we  would  like  to  stick  with  our  design  unless  a  very 
good  implementation  or  proposal  comes  out. 


Because,  with  this  infrastructure  we,  not  only  define 
a  reliability  service  but  also  a  key  component  of  an 
architecture,  that  provides  query,  caching,  filtering 
service  interfaces  to  the  application  programs, 
hiding  the  details  of  underlying  in-house  and  OMA 
structures. 

Persistency  service  may  also  be  used  by  the 
“permanent”  entity  manager  if  an  easy  to  use  service 
is  implemented.  On  the  other  hand,  extemalization 
service  can  also  be  utilized  in  order  to  achieve  the 
data  storage  in  the  cashe  with  some  modifications. 

Time  service  is  also  another  very  important  service 
in  the  overall  architecture.  Currently,  we  are 
planning  to  use  NTP  for  time  alignment.  It  gives 
sufficient  resolution  and  accuracy  for  the  console 
domain  operations  but  not  for  the  weapon-sensor 
domain.  Hence,  better  time  services  are  expected  to 
evolve  or  a  hardware  timer  may  be  utilized  in  the 
eventual  system. 

We  believe  that  it  will  take  some  time  for  the  new 
services  to  be  mature  enough  to  be  used  in  such  an 
architecture,  and  we  will  continue  to  utilize  the 
existing  ones,  and  since  they  are  modular  and  as  long 
as  we  define  the  application  program  interfaces 
correctly,  changing  the  underlying  mechanism  will 
not  be  a  big  problem. 

6.  DISCUSSION  AND  CONCLUSIONS 

In  this  paper,  after  the  evaluation  phase,  we  have  first 
presented  a  reliable  CORBA  infrastructure  based  on 
redundant  entities,  and  secondly,  presented  domain 
specific  CORBA  based  architecture  for  C2  purposes. 
We  utilized  the  ideas  in  [10],  [2]  and  our  experience 
in  this  domain. 

Although  our  main  concern  is  the  reliability  feature 
of  the  architecture,  with  the  infrastructure  we 
provide,  we  present  a  layer  which  also  supports 
query,  filtering,  typed  event  channels  and 
notification  services.  Although,  these  services  are 
handled  in  the  infrastructure  layer  in  one  entity 
object  manager,  depending  of  the  type  of  the  entity 
object,  if  the  performance  results  permits,  we  can 
easily  divide  these  different  services,  into  different 
modules  according  to  their  fuctionalities. 

We  believe  that  publish-subscribe  mechanism 
establish  a  good  infrastructure  for  large-scale 
software  systems.  CORBA,  even  with  some  its 
missing  features,  provides  support  for  this 
mechanism.  Encapsulating  the  data  as  the  entity 
objects  in  sucb  a  mechanism,  allows  the  usage  of 
object  features  for  pure  data  in  the  system,  such  as 
type-checking  and  inheritance. 
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Our  proposed  module  on  top  of  the  event  service  of 
the  CORBA  architecture  gives  us  the  opportunity  to 
introduce: 

•  reliable  programming  environment  by  using 
redundant  data, 

•  typed  event  channels  for  application  programs, 

•  better  performance  expectance  due  to  the  less 
number  of  consumer  programs,  and  effective  in¬ 
node  communications. 

•  query  mechanism  for  the  application  programs 
which  will  in  many  case  increase  the  perform¬ 
ance, 

•  an  effective  access  to  the  information  for  the 
application  programs. 

Since  the  implementation  is  not  ready  yet,  currently 
we  can  not  give  any  performance  results,  but  from 
an  architectural  point  of  view,  the  design  support 
the  idea  of  independent  modules,  utilizing  COTS 
products  to  the  extend  possible  which  causes  easy 
modifications  and  little  in-house  implementation, 
letting  the  application  designers  and  programmers 
concentrate  on  their  application  functions  for  C2 
system  domain.  We  hope  to  present  the  perform¬ 
ance  results  of  the  prototype  which  will  not  include 
the  query  and  filter  mechanisms  soon. 

After  setting  the  first  prototype,  we  would  like  to 
build  an  API  for  object-oriented  application  pro¬ 
grams.  This  will  enable  us  to  use  some  of  the  soft¬ 
ware  patterns  in  our  architecture  more  easily. 

Investigating  the  possible  connection  mechanisms 
of  agent  based  systems  into  this  architecture  is 
another  issue  that  we  consider  for  further  study. 

Using  Java  based  programs  for  especially  user- 
interface  programs  in  connection  with  CORBA 
architecture  will  support  adaptability,  maintainabil¬ 
ity  and  industry  support  features  of  the  proposed 
architecture.  Hence  this  kind  of  joint  usage  will  be  a 
matter  of  future  progress. 
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SUMMARY 

Automatic  target  cueing  and  pilot  voice  recognition  were 
integrated  into  a  single-seat  fighter  cockpit  simulator  and 
were  evaluated.  Pilots  were  required  to  fly  a  pre-planned 
route  to  an  airfield,  where  they  identified  and  designated 
for  attack,  six  tanker  aircraft  from  a  group  of  fifteen 
aircraft  that  were  parked  on  the  airfield.  During 
navigation  and  weapon  delivery  segments  of  the  mission, 
simulated  Airborne  Warning  and  Control  directed  the 
pilots  to:  1)  modify  their  flight  route,  2)  change  radio 
frequencies,  3)  respond  to  various  tasks  and  instructions, 
and  4)  attack  the  airfield.  During  half  of  the  data 
collection  sessions,  data  input  tasks  were  performed 
manually  by  the  pilots  using  an  “upfront”  keypad;  during 
the  other  half  of  the  sessions,  data  input  was 
accomplished  by  voice.  Additional  independent 
variables  were:  1)  auditory  interference — number  of 
communications  requiring  pilot  response,  and  2) 
workload — maintain  altitude  at  either  300  feet  or  1000 
feet  above  ground  level.  Objective  measures  of 
performance  for  data  input  (speed  and  accuracy)  and  for 
aircraft  control  (deviations  from  commanded  course, 
airspeed  and  altitude)  were  collected  while  pilots 
navigated  along  the  fight  route.  Objective  measures 
collected  during  ground  attack  included  speed  of  target 
designation,  total  number  of  targets  correctly  designated, 
and  stand-off  distance  from  the  airfield  at  target 
designation. 

INTRODUCTION 

Today,  two-seat  weapon  systems  are  required  to 
successfully  accomplish  the  most  demanding  ground 
attack  Air  Force  missions  because  cockpit  labor  must  be 
divided  between  a  pilot  and  weapon  system  operator  to 
mitigate  task  saturation  and  to  maintain  adequate  aircrew 
situation  awareness.  In  large  measure,  task  saturation 
and  inadequate  situation  awareness  occur  because 
current  cockpit  technologies  perpetuate  three  main 
behaviors  that  reduce  aircrew  effectiveness:  1 )  frequent 
transition  from  head-up  to  head-down  displays,  2)  cross¬ 
check  of  information  that  is  presented  on  multiple 
displays,  and  3)  mental  translation  of  two-dimensional 
display  information  into  a  three-dimensional 
environment. 

A  series  of  simulation  experiments  at  the  U.S.  Air  Force 
Research  Laboratory  (AFRL)  have  quantified  the 
advantages  of  integrating  various  human-computer 
interface  (HCI)  technologies  to  achieve  the  goals  of 
reducing  aircrew  workload,  increasing  aircrew  situation 
awareness,  and  permitting  accomplishment  of  the  most 
demanding  types  of  ground  attack  missions  with  one 
crewmember  instead  of  two.  In  early  evaluations,  HCI 
technologies  integrated  into  a  single-seat  fighter  cockpit 
simulator  were  a  helmet-mounted  display  (HMD)  with 
head-steered  sensor  capability,  a  directional-audio  threat 
cueing  system  and  large-screen,  head-down  displays. 


Results  showed  that  an  HMD  integrated  with  head- 
steered  sensor  capability  significantly  increased  pilot 
accuracy  and  speed  to  acquire  threats  and  targets 
positioned  off-axis  of  the  aircraft  centerline,  and  also 
dramatically  improved  pilot  situation  awareness  over  a 
baseline,  non-HMD  system.  A  head-down  cockpit 
configuration  with  a  centrally-located,  8”-  square, 
primary  display  and  6”  x  8”  displays  on  the  left  and 
right  promoted  more  efficient  instrument  cross-check 
and  provided  the  requisite  display  area  for  pilots  to 
accomplish  time-critical  ground  target  search  and 
identification  tasks  when  compared  to  cockpit 
configurations  that  used  larger,  but  fewer,  display 
screens  (Ref  1,2,  3). 

Background 

Most  recently,  automatic  target  cueing  and  pilot  voice 
recognition  were  integrated  and  evaluated  in  the 
simulator.  Voice  control  has  been  researched 
extensively  over  the  past  20  years.  Several  researchers 
(Ref  4)  have  proposed  voice  control  as  a  natural  and 
intuitive  method  by  which  pilots  can  communicate  with 
the  complex  interfaces  found  in  the  cockpit.  Voice 
control  could  allow  pilots  to  manage  various  sources  of 
information  more  efficiently  by  reducing  resource 
competition  (e.g.,  visual  attention  requirements),  freeing 
the  pilot’s  hands  and  eyes  to  fly  the  aircraft,  simplifying 
complex  strings  of  control  actions  through  the  use  of 
“voice  macros,”  and  taking  advantage  of  a  natural,  well- 
leamed  human  behavior — speech. 

Voice  can  also  provide  an  alternative,  simplified  control 
method  to  complex,  overloaded  hands-on  throttle  and 
stick  (HOTAS)  implementations.  According  to  an 
Advisory  Group  for  Aerospace  Research  & 
Development  published  report  (Ref  5),  cockpits  are 
becoming  increasingly  complex  because  of  the  heavy 
reliance  on  programmable,  multifunction  displays  and 
controls,  and  advances  in  communications,  data  link, 
weapon  and  sensor  system  capabilities.  With  today’s 
more  complex  cockpits,  “the  ability  to  control  aircraft 
sensor  and  weapon  functions  through  software-driven 
keys  has  increased  with  a  commensurate  increase  in 
functions  controlled  by  HOTAS.”  AGARD  concludes 
that  “HOTAS  switches  have  reached  practical  limits... 
and  will  force  future  use  of  alternative  methods  [of 
input].”  Voice  control  offers  an  alternative  method  of 
control  input  that  retains  many  of  the  benefits  provided 
by  HOTAS,  including  the  ability  to  execute  control 
actions  without  removing  hands  from  primary  flight 
controls,  and  without  distracting  visual  attention  from 
other  critical  tasks. 

Many  of  the  proposed  benefits  of  voice  control  have 
been  demonstrated  in  part-task  simulation 
environments.  Compared  with  traditional  manual 
control  methods  (e.g.,  switches,  keyboards),  voice 
control  can  improve  crew  performance  and  simplify 
operations  for  some  tasks.  For  example,  the  use  of 
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voice  control  has  been  shown  to  decrease  input  times  for 
selected  data  entry  and  concurrent  flying  tasks  (Ref  6,  7), 
increase  input  accuracy  (Ref  8),  and  permit  the  user  to 
simultaneously  attend  to  relevant  visual  tasks  (Ref  9). 
Other  studies  suggest  that  speech  recognition  technology 
can  potentially  reduce  operator  workload,  and  may 
improve  the  operator’s  ability  to  manage  multiple, 
simultaneous  tasks  in  a  high  workload  environment  (Ref 
8,  10). 

State-of-the-art  voice  systems  are  capable  of  accurately 
recognizing  continuous  speech,  which  allows  the  user  to 
speak  naturally  using  normal  inflections  and  pauses 
between  words.  The  systems  have  also  matured  to  the 
point  where  they  can  achieve  high  recognition  accuracy 
in  an  operational  cockpit  environment.  Williamson, 
Barry  and  Liggett  (Ref  1 1)  performed  flight  tests  of  a 
continuous  speech  recognition  system  in  an  OV-lOA, 
measuring  speech  recognition  accuracy  with  the  engines 
on  and  off,  and  during  continuous  3-g  turns.  They  also 
measured  recognition  accuracy  in  a  laboratoiy 
environment  for  a  baseline.  No  differences  in 
recognition  accuracy  were  found  across  the  different 
conditions,  and  at  no  time  did  accuracy  drop  below  97%. 
They  concluded  that  speech  recognition  could  be  used 
effectively  within  a  high  noise/mid-G  environment. 

Objective 

While  the  benefits  of  voice  control  have  been 
demonstrated  in  part-task  environments,  and  speech 
recognition  systems  have  matured  to  where  they  can 
perform  well  in  severe  environments,  a  voice  interface 
has  not  been  demonstrated  within  operational  contexts. 
The  overall  objective  of  this  experiment  was  to  address 
the  operational  context  issue  by;  1)  measuring  how  the 
use  of  voice  control  influenced  the  pilot’s  ability  to 
multitask  in  a  high  task  load  environment,  2) 
investigating  potential  interference  between  voice 
control  and  other  vocal  communication  tasks, 

3)  determining  how  voice  control  could  Influence 
mission  effectiveness  in  a  simulated  air  interdiction 
mission,  and  4)  assessing  the  utility  of  voice  control 
when  used  in  conjunction  with  automatic  target  cueing. 
Mechanization  of  the  voice  interface  was  based  on 
recommendations  from  several  previous  researchers  and 
fell  into  five  categories:  programming,  interrogation, 
data  entry,  switch  and  mode  selection,  and 
continuous/time-critical  action  control  (Ref  7,  12,  13). 
Speech  recognition  has  been  successfully  used  in  fighter 
aircraft  simulations  and  flight  tests  for  setting  radio 
frequencies,  setting  steerpoint  coordinates  and  weapons 
release  parameters,  and  controlling  flight  displays  (Ref 
14,  15).  The  voice  control  tasks  in  this  experiment 
covered  this  range  of  applications. 

METHOD 

Subjects 

Twelve  male  pilots  with  fighter  aircraft  experience 
participated  in  the  experiment.  Their  total  operational 
flight  times  ranged  from  a  low  of  650  hours  to  a  high  of 
3800  hours.  Their  cumulative  aircraft  experience 
included  significant  hours  in  the  A- 10,  F-4,  F-1 11,  F-15, 
F-16,  B-52,  B-2,  OV-10  and  T-38. 

Simulator 

The  experiment  was  conducted  in  the  Reconfigurable 
Single-Seat  (RS2)  simulator  in  the  Vehicle-Pilot 
Integration  Laboratory  at  Wright  Patterson  AFB.  RS2 
consisted  of  a  single-seat  cockpit  shell  with  side-console 
mounted  F-15E  stick  grip  and  throttles  and  a  21”  x  16” 
color  monitor  for  showing  the  head-down  display 
formats.  The  standard  F-15E  hands-on-throttle-and-stick 


(HOTAS)  switch  mechanization  was  slightly  modified 
to  support  the  research  requirements  of  the  experiment. 
A  Barco  Retrographics  801  projection  screen,  which 
was  located  directly  in  front  of  the  cockpit,  displayed  F- 
15E  head-up  display  symbology  and  simulated,  out-the- 
window,  forward  field  of  view  imagery.  The  RS2 
simulator  is  shown  in  Figure  1. 


Figure  1.  RS2  Simulator 


Head-Down  Display  Layout 

The  head-down  suite  consisted  of  an  8”  x  8”  top  center 
display  and  a  6.7”  x  6.7”  bottom  center  display  on 
which  sensor  information  (tactical  situation  display — 
TSD,  air-to-ground  radar,  targeting  pod  video, 
weapons)  was  provided.  Two  6”  X  8”  displays  on  either 
side  of  the  top  center  display  provided  primary  flight 
information,  system  status  information,  and 
engine/weapon  information.  The  left  6”  x  8”  display 
was  divided  into  two  separate  areas  measuring  6”  x  2” 
and  6”  x  6”  respectively.  The  6”  x  2”  area  provided 
communication  and  navigation  information  (CNI)  and 
the  6”  X  6”  area  was  a  multifunction  display  (MFD) 
capable  of  presenting  system  status  and  flight 
information.  A  keypad  was  positioned  next  to  the 
6”  X  2”  display  to  allow  the  pilot  to  manually  enter 
communication  or  navigation  data.  Figure  2  shows  the 
head-down  display  size  configuration  and  lists  the  main 
formats  available  on  each  display. 
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-Weapons 

Input 

Keypad  6.7”  x  6.7” 


-Target  video 
-Tactical  sit. 

-Radar/Weap. 

Figure  2.  Flead-Down  Display  Configuration  (Ref  16) 
Test  Operator’s  Console 

A  test  operator’s  console  was  located  to  the  side  of  the 
RS2  and  was  used  to  supervise  and  control  the 
simulation.  It  consisted  of  one  Silicon  Graphics  21” 
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monitor,  three  13”  high-resolution  monitors,  an  intercom 
system  and  a  desk/writing  area. 

Voice  Control  System 

To  enable  the  pilots  to  use  voice  commands  in  the 
cockpit,  a  Verbex  Portable  continuous  speech 
recognition  system  was  used  with  an  M- 162  microphone 
that  was  affixed  to  standard  issue  USAF  aircrew 
headsets.  The  speech  recognizer  included  training 
software  that  generated  speaker-dependent  reference 
template  files  for  the  entire  voice  control  vocabulary. 
These  individualized  templates  served  as  the  basis  for 
interpretation  of  the  pilot’s  voice  commands.  To  activate 
the  speech  recognition  system,  the  pilot  was  required  to 
pull  aft  on  the  HOTAS  microphone  switch  and  then 
speak  the  desired  voice  commands.  The  recognizer 
compared  these  inputs  to  the  stored  template,  and  then 
sent  the  interpreted  command  to  the  simulation  software. 

The  vocabulary  contained  a  total  of  91  words  and  short 
phrases,  i.e.  utterances,  and  was  developed  with  the  aid 
of  several  experienced  pilots  to  ensure  that  commonly 
accepted  terminology  was  incorporated.  To  make  the 
system  more  flexible,  and  therefore  more  robust  with 
respect  to  individual  pilot  styles,  selected  tasks  could  be 
accomplished  with  multiple  voice  commands.  For 
example,  to  slew  the  targeting  pod  to  the  next  steerpoint, 
the  pilot  could  say  ‘slew  next  steerpoint’  or  ‘slew  next’. 

Procedures 

Pilots  were  given  an  introductory  briefing,  followed  by 
“ground”  training,  simulator  practice  sessions  and, 
finally,  data  collection  sessions.  Training  and  practice 
were  designed  to  help  the  pilots  understand  their  roles 
and  responsibilities  during  data  collection  and  took 
approximately  four  hours  to  accomplish.  The  sessions 
included  control  interface  training  (voice  control  and 
manual  control  instructions)  while  “free-flying”  the 
simulator,  auto-target  cuer  practice,  and  flying  the 
mission  scenario  to  become  familiar  with  the  cockpit 
tasks  and  experimental  procedures.  For  conditions 
which  used  voice  recognition,  training  concluded  with  a 
30  minute  template-building  session  for  the  speech 
recognition  system.  In  addition,  pilots  were  briefed  on 
the  Subjective  Workload  Assessment  Technique 
(SWAT),  which  was  used  to  collect  workload  data  (Ref 
17). 

A  “part-task”  approach  was  used  for  training  the  pilots. 
During  the  experiment,  the  pilots  performed  nine  tasks 
while  flying  the  aircraft.  These  were:  changing  the 
radio  frequency,  inputting  new  latitude/longitude 
coordinates  for  a  mission  redirect,  correcting  a 
malfunction  in  the  flight  control  system,  updating  Global 
Positioning  System  (GPS)  coordinates  in  response  to  a 
GPS  failure,  responding  to  a  pop-up  threat,  aligning  the 
synthetic  aperture  radar  (SAR)  to  map  the  target  area, 
identifying  ground  targets,  designating  ground  targets, 
and  launching  weapons.  Each  of  these  tasks  was 
introduced  and  practiced  individually  prior  to  flying  the 
entire  mission  scenario  to  give  the  pilot  the  opportunity 
to  train  each  single  task  repeatedly.  Each  task  was 
practiced  until  the  subject  matter  expert  (a  pilot  on  the 
research  team)  determined  that  the  participant  was 
proficient  in  completing  that  task.  Criteria  used  in 
determining  proficiency  included  correct  responses  to 
system  changes  and  task  completion  without  cueing  from 
the  research  team. 

Data  collection  sessions  began  after  the  research  team 
had  determined  that  the  pilot  was  proficient  in  simulator 


flying  and  performing  the  various  tasks  with  each  HCI. 
Each  pilot  flew  eight  simulation  runs;  four  using  manual 
HCI  and  four  using  voice  HCI  in  a  within-subjects 
design,  i.e.  all  subjects  received  all  combinations  of  the 
independent  variables.  Presentation  order  for  the 
independent  variables  was  counterbalanced  across  pilots 
and  data  collection  trials. 

Mission  Scenario 

The  scenario  was  a  low-altitude,  night  interdiction 
mission.  The  initial  mission  objectives  were  to  destroy 
an  enemy  tank  column  and  then  proceed  to  destroy  an 
enemy  petroleum  site.  The  scenario  was  divided  into 
two  segments. 

1)  During  the  navigation  segment,  simulated  Airborne 
Warning  and  Control  radioed  new  mission  objectives  to 
the  pilot.  These  objectives  required  the  pilots  to  reroute 
to  new  Initial  Point  (IP)  and  target  coordinates  and  use 
onboard  munitions  to  destroy  enemy  aircraft  parked  at 
an  airfield.  The  pilot  was  required  to  manually  fly  the 
simulator  and  keep  the  HUD  flight  path  marker  centered 
in  the  terrain-following  (TF)  box  during  the  low  altitude 
ingress  as  the  navigation  segment  prim^  task. 
Dependent  upon  the  experimental  condition,  navigation 
segment  secondary  tasks  were  performed  using  either 
the  manual  or  voice  interface  and  required  the  pilots  to: 

1)  modify  the  flight  route  by  changing  the  steerpoint 
latitude/longitude  coordinates,  2)  change  radio 
frequencies  and  3)  respond  to  various  other  tasks  and 
instructions.  During  half  of  the  data  collection  sessions, 
the  pilots  performed  data  input  tasks  manually  by  using 
the  upfront  keypad  and  HOTAS  switches.  During  the 
other  half  of  the  sessions,  data  input  was  accomplished 
by  voice  interface — ^the  pilots  read/recited  strings  of 
latitude/longitude  coordmates,  asked  for  the  desired 
radio  frequencies,  and/or  asked  for  the  desired  display 
format.  Additional  independent  variables  during  me 
navigation  segment  were:  1)  auditory  interference — ^the 
number  of  communications  into  the  cockpit  which 
required  the  pilot  to  respond,  and  2)  workload — 
maintaining  TF  altitude  at  either  300  feet  or  1000  feet 
above  ground  level.  Objective  measures  of 
performance  for  data  input  (speed  and  accuracy)  and  for 
aircraft  control  (deviations  from  commanded  course, 
airspeed  and  altitude)  were  collected  along  the  fight 
route. 

2)  During  the  weapon  delivery  segment,  the  pilots  used 
the  head-down  displays  to  view  a  simulated  mfrared 
image  of  the  airfield  where  fifteen  aircraft  were  parked. 
As  me  pilots  flew  closer  to  the  airfield,  their  primary 
task  was  to  find  and  identify  six  tanker  aircraft  out  of  all 
the  vehicles  parked  on  the  airfield,  designate  them  as 
targets,  and  launch  weapons  against  them.  The  weapon 
delivery  segment  independent  variables  were:  1) 
manual  versus  voice  control  of  target  designation  tasks, 
and  2)  auto-target  cueing  (ATC)  versus  no  auto-target 
cueing.  With  me  manual  interface,  the  pilots  used 
conventional  HOTAS  slew  and  designation  switches  to 
assign  weapons  to  targets;  with  the  voice  interface,  the 
pilots  spoke  the  appropriate  commands  to  cue  and 
designate  the  targets  for  weapons  assignment.  For  the 
trials  with  ATC,  a  symbol  (cncle)  and  a  number  (1 
through  15)  highlighted  each  potential  target  (Ref  18). 
ATC  also  provided  a  “proximity  highlight*’  feature  that 
allowed  the  pilot  to  manually  designate  (using  HOTAS) 
without  havmg  to  center  the  targeting  pod  cursor  on  a 
specific  target.  The  cuer  symbology  of  the  target 
closest  to  the  targeting  pod  cursor,  at  any  given  time, 
was  highlighted.  This  mdicated  that  location 
coordinates  for  the  highlighted  target  would  be  sent  to 
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on-board,  directed-attack  munitions  if  a  designation  was 
performed.  Figure  3  illustrates  the  target  designation 
task  with  ATC;  target  7  is  proximity  highlighted. 

Without  ATC  (Figure  4),  no  circles  or  numbers  were 
shown  around  any  of  the  aircraft.  The  pilot  slewed  the 
target  pod  cursor  over  the  desired  target  and  then  pressed 
the  designate  button.  There  was  no  proximity  highlight 
feature;  pilots  positioned  the  cursor  over  some  portion  of 
the  desired  target  and  then  designated.  Once  designated, 
a  square  with  a  dot  in  the  center  marked  the  position  of 
the  designation. 
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Figure  3.  Target  Designation  Symbology  with 
Auto-Target  Cuer  (ATC) 


Figure  4.  Target  Designation  Symbology  without 
Auto-Target  Cuer  (ATC) 


For  weapon  delivery,  objective  measures  of  performance 
included  the  speed  with  which  pilots  could  designate 
targets,  the  total  number  of  targets  correctly  designated. 


and  the  pilot’s  distance  from  the  airfield  at  the  time  of 
target  designation.  Table  1  presents  an  experimental 
design  overview  including  the  mission  segments  and 
their  respective  tasks  and  Independent  variables. 

Hypotheses 

In  general,  the  use  of  speech  recognition  was  expected 
to:  1)  improve  system  effectiveness  by  decreasing  the 
time  required  for  the  pilot  to  perform  mission  re-route 
under  high  TF  Task  Load  during  navigation  and  by 
increasing  the  number  of  bombs  released  per  pass 
during  the  weapon  delivery  segment,  2)  increase  pilot 
ability  to  multi-task  by  decreasing  the  time  required  to 
complete  the  navigation  secondary  tasks  and  the 
weapon  delivery  primary  tasks,  and  3)  reduce  pilot 
workload  by  providing  a  more  natural  and  intuitive 
interface  when  compared  to  the  manual  interface. 

Tablet.  Evaluation  Overview 

Mission  Segment/Tasks  Independent  Variables 

Navigation 

1.  Primary  Task: 

•  manually  fly  in 
terrain-following 
(TF)  mode 

2.  Secondary  Tasks: 

•  monitor/respond  to 
radio 

•  change  radio 
frequencies 

•  troubleshoot 
failures 

•  display  reroute  info 

•  input  reroute  data 

Weapon  Delivery 

1 .  Primary  Tasks: 

•  detect,  acquire,  and 
designate  multiple 
targets  with  and 
without  auto-target 
cuer 

•  release  weapons 


RESULTS 

Summary 

Navigation  segment  results  showed  that  voice 
recognition  significantly  improved  the  speed  and 
accuracy  of  pilot  data  input  during  re-route  when 
compared  to  manual  input.  Weapon  delivery  segment 
results  showed  that  pilot  performance  was  significantly 
improved  by  integrating  auto-target  cueing  features  with 
voice  recognition  when  compared  to  the  manual, 
throttle-mounted  switches.  In  fact,  in  all  but  one  of  the 
voice  interface  cases,  pilots  were  able  to  correctly 
designate  all  six  of  the  tanker  aircraft  in  a  single  pass  on 
the  airfield  at  significantly  greater  distances  from  the 
airfield  than  when  they  used  the  manual  interface. 

Navigation  Segment 

Specific  objectives  of  the  navigation  segment  were: 

1 .  Determine  the  effects  of  voice  control  on  the  pilot’s 
ability  to  complete  secondary  tasks  while  performing 
the  primary  TF  flying  task. 


1.  Control  Type: 

•  manual 

•  voice 

2.  TF  Task  Load: 

•  low:  1000  feet  AGL 

•  high:  300  feet  AGL 

3.  Auditory  Task  Load: 

•  low:  lead  aircraft  + 

1  wingman 

No  response  required 

•  high:  lead  aircraft  + 

4  wingmen 
Responses  required 


1 .  Control  Type: 

•  manual 

•  voice 

2.  Auto-Target  Cuer: 

•  present 

•  absent 
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2.  Determine  the  word  error  rate  of  the  speech 
recognition  system  under  simulated  operational 
conditions. 

3.  Determine  pilot  acceptability  of  the  mechanization 
and  effectiveness  of  the  speech  recognition  system. 

It  was  hypothesized  that  the  voice  interface  would  not 
provide  significant  improvement  in  secondary  task 
performance  compared  to  the  manual  interface  during 
low  manual  TF  Task  Load  conditions,  especially  under 
high  Auditory  Task  Load  conditions.  However,  for  high 
TF  Task  Load,  a  significant  improvement  in  secondary 
task  performance  was  expected  with  the  voice  interface. 
It  was  further  hypothesized  that  this  performance 
improvement  would  decrease  or  become  insignificant 
under  high  Auditory  Task  Loading.  The  accuracy  of  the 
speech  recognition  system  was  expected  to  be  sufficient 
so  as  not  to  hinder  task  performance. 

To  measure  these  expectations,  objective  Measures  of 
Effectiveness  (MOE)  and  Measures  of  Performance 
(MOP),  and  subjective  Measures  of  Workload  (MOW) 
and  questionnaire  data  were  collected  and  analyzed  for 
the  navigation  segment.  Descriptions  follow: 

Navigation  Measure  of  Effectiveness 

1.  Time  to  complete  the  secondary  task:  Defined  as 
the  time  interval  which  started  at  the  onset  of  a 
secondary  task  and  ended  when  all  necessary  functions 
had  been  completed. 

Navigation  Measures  of  Performance 

1 .  Flight  path  control:  Commanded  flight  path,  course 
deviation,  and  groundspeed  data  were  collected  for  the 
manual  TF  task  to  determine  the  acceptability  of  each 
data  collection  run.  Pilots  were  required  to  keep  the 
flight  path  marker  within  the  TF  box  on  the  HUD, 
maintain  the  course  displayed  on  the  TSD,  and  maintain 
groundspeed  of  480  knots.  For  data  analysis.  Root  Mean 
Square  Error  (RMSE)  was  calculated  for  each  parameter. 
(Note:  Tolerance  bands  for  commanded  flight  path, 
course  and  groundspeed  deviation  were  established  to 
ensure  that  the  pilots  treated  the  flying  task  as  primary. 

If,  while  performing  a  secondary  task,  a  pilot’s  flight 
time  outside  of  these  tolerance  bands  exceeded  10%  of 
the  total,  the  data  collection  trial  was  repeated.) 

2.  Communication  accuracy:  Pilots  were  required  to 
answer  radio  calls  under  high  Auditory  Task  Loads. 
Response  accuracy  was  monitored  by  a  subject  matter 
expert  to  determine  if  performance  was  commensurate 
with  procedures  that  would  be  followed  during  actual 
“check-ride”  situations.  Although  this  “commensurate¬ 
ness”  was  a  subjective  measure,  the  goal  was  to  ensure 
that  the  pilots  responded  in  a  manner  to  signify  that  the 
message  was  heard  and  understood. 

3.  Speech  recognition  error  rate:  Word  recognition 
error  rate  was  collected  to  evaluate  system  performance. 

Navigation  Measures  of  Workload 

1.  The  Subjective  Workload  Assessment  Technique 
(SWAT)  was  used  to  collect  pilot  workload  data.  During 
navigation,  three  separate  SWAT  measurements  were 
collected  at  unobtrusive  points  along  the  route  and  the 
resultant  SWAT  values  were  analyzed  using  the  sum- 
percent  method  (Ref  19). 


Navigation  Questionnaire  Data 
1 .  Subjective  data  regarding  voice  interface 
effectiveness,  acceptability,  and  mechanization  was 
collected  via  a  post-mission  interview  and  a  post-study 
questionnaire. 

Navigation  Analysis 

Statistical  comparisons  of  the  performance  levels  and 
workload  ratings,  using  Multivariate  Analysis  of 
Variance  (MANOVA)  techniques,  were  performed 
against  the  three  independent  variables:  Control  Type, 
TF  Task  Load,  and  Auditory  Task  Load.  Replication 
was  collapsed  across  all  conditions  and  post  hoc 
Analyses  of  Variance  (ANOVA)  and  Tukey’s  multiple 
pairwise  comparisons  were  performed  on  the  significant 
effects  to  identify  the  nature  of  the  statistical 
differences.  For  the  questionnaire  ratings,  frequencies 
and  averages  were  computed  and  non-parametric 
statistical  tests  were  performed  to  compare  differences 
in  questionnaire  response  means.  The  SWAT  values 
were  analyzed  using  the  sum-percent  method  (Ref  19), 
which  combines  the  traditional  workload  elements 
associated  with  time,  mental  effort,  and  psychological 
stress  into  a  scale  from  zero  to  one-hundred. 

Navigation  Results 

For  the  mission  rerouting  task,  the  main  effect  of 
Control  Type  was  significant  (F  [1,1 1]  =  6.77,  p  <  .05). 
By  using  the  voice  interface,  pilots  completed  the 
rerouting  task  almost  14  seconds  quicker  that  when 
using  the  manual  interface. 

Although  all  of  the  flight  performance  parameters  met 
the  predefined  performance  criteria,  the  analysis 
showed  a  significant  two-way  interaction  for  Control 
Type  X  Auditory  Task  Load  (F  [1,11]  =  7.01,  p  <  .05). 
Post  hoc  tests  of  this  two-way  interaction  showed  that: 

1)  during  low  Auditory  Task  Loading,  Control  Type 
had  no  effect  on  TF  performance  during  rerouting,  but 

2)  during  high  Auditory  Task  Loading,  pilots  more 
accurately  followed  the  TF  cue  when  using  the  voice 
interface  to  accomplish  rerouting  than  when  they  used 
the  manual  Interface. 

More  importantly,  the  three-way  interaction  for  Control 
Type  X  Auditory  Task  Load  x  TF  Task  Load  was  also 
significant  (F  [1,11]  =  4.72,  p  <  .05).  It  showed  that:  1) 
during  low  Auditory  Task  Loading  and  regardless  of 
Control  Type,  pilots  flew  the  TF  task  more  precisely  in 
the  low  TF  Task  Load  condition  than  in  the  high  TF 
Task  Load  condition  during  rerouting  (Figure  5),  and  2) 
during  high  Auditory  Task  Loading,  using  the  voice 
interface  for  rerouting  decreased  TF  deviations 
compared  to  the  manual  interface  when  TF  Task 
Loading  was  high,  but  had  no  effect  on  TF  deviations 
when  TF  Task  Loading  was  low  (Figure  6). 
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voice  interface  was  rated  as  highly  effective  for 
accomplishing  the  task  (average  rating  =  4.8). 

Figure  7  illustrates  the  incidence  of  three  types  of 
speech  recognition  errors  for  the  navigation  segment; 
insertions,  deletions,  and  substitutions.  Insertions  are 
errors  where  the  voice  system  inserts  commands  that  the 
pilot  did  not  speak.  Deletions  are  errors  where  the 
voice  system  did  not  recognize  what  the  pilot  correctly 
spoke  and  took  no  action.  Substitutions  are  errors 
where  the  voice  system  misrecognized  what  the  pilot 
spoke  and  executed  the  wrong  command.  The  figure 
illustrates  the  voice  interface  error  rate  during  the 
navigation  segment  for  nine  of  the  twelve  pilots — data 
from  three  pilots  were  not  recorded.  The  figure  shows 
voice  interface  error  rate  overall  averaged  2.2%. 


Control  Type 


Figure  5.  Terrain  Following  (TF)  Deviation  during 
Navigation  as  a  Function  of  Control  Type 
With  Low  Auditory  Task  Load 
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Figure  6.  Terrain  Following  (TF)  Deviation  during 
Navigation  as  a  Function  of  Control  Type 
With  High  Auditory  Task  Load 


Many  of  the  errors  were  “single-repeat”  errors.  These 
were  recognition  errors  that  were  resolved  with  a  single 
repeat  of  the  command.  Most  of  these  single  repeat 
errors  could  have  been  avoided  through  more  extensive 
voice  template  training,  so  another  analysis  was 
performed  on  an  adjusted  data  set  in  which  these  errors 
were  removed.  This  analysis  revealed  that  voice 
interface  overall  average  error  rate  drops  to  0.6%  and  is 
also  shown  in  the  figure. 
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Figure  7.  Voice  Interface  Error  Rate  (%)  for  the 
Navigation  Segment 


In  regard  to  workload  during  rerouting,  there  was  a 
significant  two-way  interaction  for  Control  Type  x  TF 
Task  Load  (F  [1,11]  =  4.84,  p  <  .05).  Post-hoc  tests 
showed  that:  1)  during  high  TF  Task  Loading,  when  the 
voice  interface  was  used  to  accomplish  mission 
rerouting,  it  decreased  pilot  workload  by  31%  compared 
to  using  the  manual  interface,  and  2)  during  low  TF  Task 
Loading,  Control  Type  had  no  effect  on  rerouting. 
During  high  TF  Task  Loading,  the  pilots’  average 
SWAT  ratings  to  reroute  the  mission  dropped  from  a  39 
with  the  manual  interface  to  a  27  with  the  voice 
interface,  whereas  during  low  TF  Task  Loading,  pilots’ 
average  SWAT  ratings  remained  virtually  constant  at  34. 

Pilot  questionnaire  responses  confirmed  that  the  voice 
interface  was  significantly  more  effective  in  completing 
the  mission  reroute  than  &e  manual  interface.  The 
effectiveness  scale  ranged  from  1  (Poor)  to  5  (Very 
Good).  Across  pilots,  the  manual  interface  was  rated  as 
neither  a  hindrance  nor  an  advantage  for  accomplishing 
the  reroute  task  (average  rating  =  3.2).  Across  pilots,  the 


Other  tasks  during  the  navigation  segment  in  which  the 
voice  and  manual  interfaces  were  compared  included: 

1)  correcting  system  malfunctions,  and  2)  changing 
radio  frequency  chaimels  to  place  radio  calls.  For 
changing  radio  frequencies — a  task  similar  in  nature  to 
rerouting  in  that  a  string  of  alphanumerics  was  entered 
into  the  simulator,  the  voice  interface  improved  pilots’ 
speed  and  accuracy  to  accomplish  the  task  compared  to 
the  manual  interface  (F  [1,1 1]  =  7.63,  p  <  .05). 

However,  the  improvement  was  less  than  what  the  voice 
interface  provided  for  the  mission  rerouting  task.  For 
one  system  malfunction — flight  control  pitch  fault — 
voice  and  manual  interfaces  demonstrated  no 
differences  for  accomplishing  the  pitch  fault  reset  task; 
for  another  malfunction — Global  Positioning  System 
(GPS)  failure — ^the  voice  interface  increased  the  time  to 
complete  the  GPS  failure  task  compared  to  the  manual 
interface.  Possible  reasons  for  these  results  will  be 
explained  in  the  discussion  section. 

Weapon  Delivery  Segment 

Specific  objectives  of  the  weapon  delivery  segment 

were: 
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1 .  Determine  the  effects  of  voice  control  on  the  pilot’s 
detection,  acquisition,  and  weapon  delivery  performance 
with  and  without  auto-target  cuer  (ATC)  while  attacking 
multiple  targets  during  a  single  pass. 

2.  Determine  the  word  error  rate  of  the  Verbex 
Portable  speech  recognition  system  under  simulated 
operational  conditions. 

3 .  Determine  pilot  acceptability  of  the  mechanization 
and  effectiveness  of  the  speech  recognition  system. 

It  was  hypothesized  that  incorporating  the  voice  Interface 
alone  would  not  provide  significant  performance  or 
workload  improvement  over  the  manual  interface. 
However,  when  mechanized  with  an  ATC,  the  voice 
interface  was  expected  to  provide  significant 
performance  and  workload  improvement  over  manual 
interface  alone  and  manual  interface  with  ATC. 

To  measure  these  expectations,  similar  MOEs,  MOPs, 
and  MOWS  to  those  identified  for  the  navigation 
segment  were  collected  and  analyzed.  Descriptions 
follow: 

Weapon  Delivery  Measures  of  Effectiveness 

1 .  Range  to  first  target:  Defined  as  the  distance  from 
the  aircraft  to  the  center  of  the  airfield  at  the  time  of  the 
first  correct  designation. 

2.  Range  to  last  target:  Defined  as  the  distance  from 
the  aircraft  to  the  center  of  the  airfield  at  the  time  of  the 
last  correct  designation. 

3.  Target  designation  rate:  The  amount  of  time  spent 
designating  per  the  number  of  correct  target 
designations. 

4.  Target  designation  accuracy:  The  total  number  of 
correct  targets  designated  per  total  number  of  designated 
targets  for  each  trial. 

Weapon  Delivery  Measures  of  Performance 

1 .  Flight  path  control:  Course  and  groundspeed  data 
were  collected  to  evaluate  aircraft  control  performance. 
Pilots  were  instructed  to  maintain  the  course  displayed 
on  the  TSD  and  maintain  groundspeed  of  480  knots. 
RMSE  was  calculated  for  each  parameter. 

2.  Speech  recognition  error  rate:  Word  recognition 
error  rate  was  collected  to  evaluate  system  performance. 

Weapon  Delivery  Measures  of  Workload 

1 .  As  in  the  navigation  segment,  SWAT  was  used  to 
collect  pilot  workload  data.  One  SWAT  measurement 
was  taken  per  pilot  after  completion  of  the  weapon 
delivery  segment. 

2.  On  the  post-study  questionnaire,  pilots  were  also 
asked  about  voice  interface  effectiveness  and 
mechanization  as  applied  to  weapon  delivery. 

Weapon  Delivery  Analysis 

Statistical  comparisons  of  the  performance  levels  and 
workload  ratings  using  MANOVA  were  performed 
against  the  two  independent  variables:  Control  Type  and 
Auto-target  Cueing.  Replication  was  collapsed  across  all 
conditions  and  post  hoc  Analyses  of  Variance  (ANOVA) 
and  Tukey’s  multiple  pairwise  comparisons  were 
performed  on  the  significant  effects  to  identify  the  nature 
of  the  statistical  differences.  For  questionnaire  ratings, 
frequencies  and  averages  were  computed  and  non- 
parametric  statistical  tests  were  performed  to  compare 
differences  in  questionnaire  response  means.  As  in  the 


navigation  segment,  the  SWAT  values  were  analyzed 
using  the  sum-percent  method  (Ref  19). 

Weapon  Delivery  Results 

For  weapon  delivery,  the  main  effect  of  Control  Type 
was  significant  (F  [1,11]  =  5.68,  p  <  .05)  for  airspeed 
deviation:  MANOVA  results  showed  that  pilots  had 
greater  deviations  from  the  designated  airspeed  of  480 
Imots  when  using  the  voice  interface  for  target 
designation  than  when  using  the  manual  interface. 

MANOVA  analysis  also  showed  a  significant  two-way 
interaction  for  Control  Type  x  ATC  (F  [1,1 1]  =  4.79,  p 
<  .05)  for  the  time  required  to  designate  each  target. 
Post  hoc  analysis  showed  that:  1)  without  ATC, 
Control  Type  had  no  significant  effect  on  target 
designation  rate,  2)  with  ATC,  target  designation  rate 
was  significantly  faster  with  the  voice  interface  than 
with  the  manual  interface,  and  3)  target  designations 
were  accomplished  more  quickly  with  ATC  than 
without  ATC  regardless  of  the  Control  Type.  The 
weapon  delivery  two-way  interaction  is  depicted  in 
Figure  8. 


Figure  8.  Target  Designation  Rate  during  Weapon 
Delivery  as  a  Function  of  Control  Type  With 
and  Without  Auto-Target  Cueing 


MANOVA  results  also  showed  a  significant  Control 
Type  X  ATC  interaction  for  workload  (F  [1,11]  =  4.73, 
p<.05).  Post  hoc  tests  showed  that:  1)  without  ATC, 
Control  Type  had  no  effect  on  pilot  workload,  and  2) 
with  ATC,  average  pilot  workload  ratings  for  using  the 
voice  interface  were  22%  lower  (average  rating  =  46) 
than  those  for  the  manual  interface  (average  rating  = 

59).  The  weapon  delivery  workload  results  are  depicted 
m  Figure  9  (next  page). 

Results  of  the  weapon  delivery  questionnaire  data  were 
consistent  with  the  navigation  segment  questionnaire 
data.  During  weapon  delivery,  pilots  rated  the  voice 
interface  combined  with  the  auto-target  cuer  as  most 
effective  for  identifying  targets  when  compared  to  the 
manual  or  voice  interfaces  alone  or  the  manual  interface 
combined  with  ATC.  For  designating  targets,  pilots 
rated  both  control  types  effective  regardless  of  whether 
or  not  ATC  was  used,  but  they  rated  their  performance 
most  effective  and  gave  their  highest  ratings  for  the 
voice  interface  combined  with  ATC. 
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Figure  9.  Pilot  Workload  during  Weapon  Delivery 
as  a  Function  of  Control  Type  With  and 
Without  Auto-Target  Cueing 


Figure  10  illustrates  the  speech  recognition  error  rates 
for  weapon  delivery.  Insertions,  deletions,  and 
substitutions  (see  navigation  segment  for  descriptions  of 
these  error  types)  are  shown  for  nine  of  the  twelve  pilots. 
The  voice  interface  had  an  overall  error  rate  during 
weapon  delivery  of  3.9%. 

The  figure  shows  that  subject  two  contributed  a 
relatively  large  number  of  insertion  errors.  The 
reduction  in  the  insertion  error  rate  for  subsequent 
subjects  was  due  to  a  change  made  in  the  voice  interface 
mechanization.  This  will  be  referenced  in  the  Discussion 
section.  When  the  single-repeat  errors  were  removed 
from  the  data  set,  the  adjusted  overall  mean  error  rate 
dropped  to  1.9%. 


Subject 

Figure  10.  Voice  Interface  Error  Rate  (%)  for  the 
Weapon  Delivery  Segment 


DISCUSSION 


Navigation  Segment 

It  was  hypothesized  that  voice  control  would  enhance 
secondary  task  performance  when  the  pilot  was  under 
the  visual  and  manual  demands  imposed  by  the  high 
manual  Terrain  Following  Task  Loading.  It  was  also 
hypothesized  that  the  high  Auditory  Task  Loading  would 
interfere  with  the  pilot’s  ability  to  perform  the  secondary 


tasks  using  voice  control.  A  secondary  purpose  of  the 
experiment  was  to  determine  if  voice  control  could  act 
as  a  viable  substitute  for  some  manual  tasks,  so  it  was  of 
interest  when  no  differences  in  performance  were  found 
between  voice  and  manual  control  conditions. 

The  results  showed  that  voice  control  decreased  the 
amount  of  time  it  took  to  complete  the  mission  reroute 
task  and  the  radio  frequency  change  task.  Mission 
reroute  results  further  showed  that  voice  control 
reduced  workload  under  high  TF  Task  Loading 
conditions.  Finally,  the  subjective  data  analysis  showed 
that  pilots  rated  the  voice  control  interface  as 
significantly  more  effective  than  the  manual  control 
interface  for  mission  reroute  and  radio  frequency 
change  tasks.  At  least  two  factors  may  have  contributed 
to  these  results.  First,  the  voice  interface  enabled  the 
pilots  to  multitask  more  efficiently  than  was  possible 
with  the  manual  control  interface.  This  is  consistent 
with  the  hypothesis  that  voice  control  reduces 
competition  for  human  resources  such  as  visual 
attention  or  manual  control  actions.  When  applied 
appropriately,  voice  control  may  aid  in  distributing  task 
demands  across  the  operator’s  visual,  manual,  and 
voice^uditory  resources,  thereby  aiding  the  pilot  in 
dividing  attention  among  the  concurrent  tasks.  Second, 
voice  control  simplified  the  data  entry  tasks  by  reducing 
the  number  of  control  actions  required  to  complete  the 
task.  \^en  these  tasks  were  performed  manually, 
mission  rerouting  required  39  and  radio  frequency 
changes  required  7  key  strokes  to  complete.  Voice 
implementation  required  only  8  and  1  utterances 
respectively.  This  reduction  was  achieved  by  grouping 
many  of  the  serial  keystrokes  into  spoken  strings. 

The  results  also  showed  that  the  voice  interface 
increased  the  time  to  complete  the  GPS  failure  task,  but 
had  no  effect  on  pilot  performance  of  the  Pitch  Fault 
task.  The  characteristics  of  these  tasks  provide  some 
insight  into  these  results.  In  both  tasks,  manual  switch 
hits  were  replaced  with  voice  commands  on  a  one-for- 
one  basis.  Also,  control  feedback  for  each  switch  hit 
was  provided  visually  for  the  manual  interface  and 
visually  and  auditorally  for  the  voice  interface.  Several 
sources  of  time  delay  are  possible  for  the  voice 
implementation.  First,  the  pilot  was  required  to  press  a 
switch  to  talk  to  the  voice  system,  and  follow  this  action 
with  a  voice  command.  At  least  two  actions  were 
required,  then,  when  a  single  manual  switch  hit  was 
replaced  with  a  voice  command.  Second,  it  often  takes 
more  time  to  speak  a  one  or  two  word  command  than  to 
manually  activate  a  switch,  especially  if  the  switch  is 
immediately  accessible,  sucb  as  in  HOTAS.  Third,  the 
voice  messages  which  were  provided  as  feedback  to 
voice  commands  were  serial  in  nature  and  took  more 
time  to  present  than  the  visual  feedback,  which  was 
displayed  immediately.  If  pilots  chose  to  wait  for  the 
verbal  feedback  rather  than  rely  on  the  visual  feedback, 
their  task  times  would  have  necessarily  been  increased. 
Although  voice  feedback  during  the  GPS  failure  was 
not  extensive,  it  is  possible  that  it  contributed  to  the 
difference.  Finally,  because  speech  recognition  systems 
are  not  100%  accurate,  one  would  expect  to  see  more 
errors  with  a  voice  interface  than  with  a  manual 
interface,  even  if  the  correct  input  (or  voice  command) 
is  made  by  the  pilot.  In  fact,  the  voice  command 
“Mark,”  which  was  used  during  the  GPS  failure  task, 
yielded  the  highest  observed  deletion  error  rate  (20%). 
The  need  for  pilots  to  repeat  this  command  undoubtedly 
increased  the  average  task  completion  time. 

While  these  four  potential  sources  of  time  delay  may 
increase  task  performance  time,  the  results  also  show 
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that  they  will  not  always  do  so  (such  as  with  the  Pitch 
Fault  task).  It  is  clear  that  the  effect  of  these  time  delays 
is  highly  dependent  upon  the  specific  nature  of  the  task 
and  the  pilot’s  strategy  in  performing  the  task,  and,  in 
some  cases,  may  be  so  small  as  to  be  undetectable  and 
inconsequential. 

Although  statistically  significant  differences  were  found 
in  the  analysis  of  aircraft  flying  performance,  all  pilots 
were  able  to  stay  within  the  prescribed  performance 
criteria.  With  the  exception  of  course  deviation  during 
the  GPS  failure  task,  all  significant  differences  show  that 
the  voice  interface  yielded  performance  better  than  or 
equal  to  the  manual.  This  result  further  supports  the 
notion  that  voice,  when  implemented  properly,  can 
facilitate  pilot  performance  in  a  multitasking 
environment  by  reducing  resource  competition.  More 
specifically,  using  the  voice  system  may  have  enabled 
the  pilots  to  allocate  more  attention  to  critical  flight 
displays  (i.e.,  the  terrain  following  guidance  symbology) 
by  reducing  the  visual  demand  associated  with  the 
secondary  tasks. 

An  unexpected  result  of  the  navigation  segment  was  the 
lack  of  significant  effects  for  Auditory  Task  Loading. 
The  performance  data  indicated  that  voice  control  was 
equally  effective  (or  ineffective)  regardless  of  the  level 
of  Auditory  Task  Load.  However,  several  pilots 
commented  in  the  questionnaires  that  using  voice  had  the 
potential  to  interfere  with  their  ability  to  monitor  radio 
communications. 

Finally,  many  of  the  statistical  comparisons  performed 
on  the  performance  and  subjective  data  from  the 
navigation  segment  yielded  no  statistically  significant 
differences.  Overall,  then,  the  navigation  results  indicate 
that  voice  is  a  viable  control  method  that  can  serve  to 
compliment  or  relieve  an  overloaded  manual  control 
interface.  The  results  also  indicate  that,  when 
implemented  to  take  advantage  of  the  automated 
processing  characteristics  of  voice,  voice  control  can 
enhance  performance.  Conversely,  voice  does  not 
improve — and  may  actually  hinder — ^performance  when 
used  for  single  switch-hit  tasks  that  are  easily 
accomplished  via  HOTAS. 

Weapons  Delivery  Segment 

It  was  hypothesized  that  incorporating  the  voice  interface 
by  itself  would  not  provide  significant  performance  or 
workload  improvement  over  the  manual  interface. 
However,  when  mechanized  with  an  ATC,  the  voice 
interface  was  expected  to  provide  significant 
performance  and  workload  improvement  compared  to 
the  manual  interface  alone  and  the  manual  interface  with 
ATC. 

The  results  verified  the  hypotheses  and  showed  that  the 
pilots  were  able  to  designate  targets  more  quickly  using 
voice  control  coupled  with  the  ATC  than  with  manual 
control  coupled  with  the  ATC.  Further,  pilots  reported  a 
decrease  in  workload  when  using  the  voice  versus 
manual  interface  in  combination  with  the  ATC. 

These  differences  can  be  attributed  to  the  voice 
mechanizations  of  the  cueing  and  target  designation 
functions.  The  cueing  function  allowed  the  pilot  to 
automatically  slew  the  targeting  pod  onto  any  target  by 
saying  “cue”  and  then  the  target  number.  Once  the  target 
was  cued,  it  could  be  designated  by  commanding 
“designate  (target  number)”;  if  desired,  the  pilot  could 
designate  up  to  three  targets  at  a  time  using  this  method, 
e.g.  “designate  targets  x,  y,  z”.  This  mechanization 


provided  immediate,  “random”  access  to  desired  targets. 
With  the  manual  interface,  on  the  other  hand,  pilots 
were  required  to  slew  the  target  pod  across  the  airfield 
to  locate  the  desired  target.  Once  located,  additional 
slewing  was  required  to  precisely  position  the  cursor  on 
the  desired  target  (without  ATC)  or  slew  the  cursor  such 
that  the  desired  target  was  highlighted  (with  ATC 
proximity  highlight  feature).  The  additional  slewing 
associated  with  the  manual  interface  was  significantly 
more  time-consuming  than  the  voice  interface  method. 

After  removing  subject  two’s  data,  the  largest  sources  of 
error  during  weapon  delivery  were  substitutions  and 
deletions.  Although  both  types  of  error  only  occurred 
0.4%  of  the  time,  it  can  still  be  accounted  for.  Closer 
examination  of  the  data  showed  that  the  substitution 
errors  occurred  when  designating  multiple  targets.  The 
voice  control  mechanization  prior  to  subject  t&ee 
allowed  pilots  to  designate  up  to  six  targets  in  a  single 
utterance.  It  was  found  that  the  voice  recognizer  was 
confusing  phrases  such  as  “Designate  Fourteen,  Eight, 
One”  as  “Designate  Eight,  Four,  Ten,  Eight,  One”.  It 
would  designate  five  targets  instead  of  three.  When  the 
system  was  limited  to  three  designations,  it  “knew”  that 
the  latter  phrase  was  invalid.  Further  software 
modifications  also  enabled  the  voice  recognizer  to 
eliminate  certain  words  as  possible  matches  due  to 
context.  Since  the  pilot  was  designating,  the  voice 
recognizer  could  exclude  any  words  not  associated  with 
designating  (such  as  “Flares”  being  substituted  for 
“Five”).  This  is  an  example  of  the  Verbex’s  ability  to 
take  advantage  of  selective  vocabularies. 

Deletion  errors  during  weapons  delivery  were  attributed 
to  the  voice  recognizer  not  being  able  to  match  the 
verbal  command(s)  to  any  commands  in  the  template. 

In  fact,  the  deletion  error  rate  was  the  predominant 
cause  of  errors  throughout  the  entire  experiment.  These 
deletion  errors  can  be  substantially  decreased  with  more 
aggressive  and  repetitive  training  of  the  voice  templates 
for  “problem  words”. 

LESSONS  LEARNED 

This  experiment  offered  a  variety  of  lessons  learned  for 
future  voice  control  implementations.  The  lessons 
learned  are  based  upon  experimenter  observations  and  a 
synthesis  of  the  objective  results. 

1)  An  extensive  vocabulary  was  developed  with  the  aid 
of  several  pilot  subject  matter  experts,  who  attempted  to 
make  it  as  intuitive  as  possible.  However,  commonly 
used  terminology  for  some  functions  and  systems  vary 
across  different  aircraft  platforms,  and  therefore  across 
pilots.  To  account  for  this,  designed-in  software 
flexibility  enabled  the  pilots  to  use  terminology  that  was 
most  familiar  to  them.  But  even  with  this  precaution, 
several  cases  were  observed  where  pilots  had  difficulty 
remembering  the  command  or  syntax  associated  with 
some  functions.  In  these  cases,  performance  obviously 
suffered,  and  the  limitations  of  the  vocabulary  detracted 
from  the  intuitiveness  of  the  interface.  To  minimize 
such  effects,  voice  commands  should  be  simple, 
consistent  across  different  functions,  and  easy  for  the 
user  to  remember. 

2)  For  many  of  the  tasks,  the  voice  interface  was 
implemented  as  a  one-for-one  replacement  of  manual 
switch  actions.  This  was  done  to  facilitate  an  “apples  to 
apples”  comparison  of  voice  and  manual  control 
techniques,  and  to  allow  an  assessment  of  the  ability  of 
voice  control  to  reduce  resource  competition  in 
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multitasking  conditions.  The  design  team  recognized 
that  this  approach  did  not  take  advantage  of  using  voice 
“macros”  to  simplify  the  interface.  Voice  macros  offer 
the  potential  of  executing  a  long  string  of  switch  actions 
through  a  single  voice  command.  The  performance 
advantages  of  macros  are  obvious,  and  should  be 
considered  during  voice  interface  design. 

3)  When  integrated  into  a  cockpit,  voice  control  should 
be  incorporated  only  as  a  redundant  method.  This  will 
allow  pilots  to  choose  the  most  cornpatible  control 
method  for  a  given  situation  and  will  prevent 
interference  between  voice  control  and  radio 
communications  tasks.  It  also  offers  the  pilot  an 
alternative  in  the  event  of  degraded  speech  recognition 
performance. 

4)  Using  voice  control  for  single  HOTAS  switch  actions 
should  be  approached  with  caution.  As  shown  here, 
performance  may  be  degraded  for  such  applications  for  a 
variety  of  reasons.  On  the  other  hand,  voice  can  be  a 
good  alternative  to  HOTAS  when  the  manual  control 
interface  is  overloaded  and  during  heavy  multitasking 
situations.  Replacement  of  single  manual  switch  hits 
with  voice  commands  should  be  considered  when  these 
conditions  exist. 

5)  Initial  integration  of  voice  feedback  in  response  to 
voice  commands  was  extensive  during  pre-experimental 
check-out.  Feedback — such  as  repeating  the  “entered- 
in”  radio  frequencies  and  steerpoint  coordinates — ^proved 
operationally  ineffective  and  added  no  value  to  fye 
interface.  In  fact,  the  initial  voice  feedback  design 
substantially  increased  task  completion  times  for  voice 
control  because  pilots  had  to  wait  for  the  voice  feedback 
before  issuing  the  next  command.  Voice  feedback 
should  be  used  only  when  it  increases  pilot  performance 
and/or  situation  awareness. 

6)  Finally,  speech  recognition  error  rate  can  be 
decreased  by  using  selective  vocabularies.  For  example, 
when  changing  radio  frequencies,  the  voice  recognizer 
should  reference  a  separate  template  with  that  specific 
vocabulary.  Speech  recognition  error  rate  can  also  be 
decreased  through  repetitive  training  of  the  voice 
templates  of  “problem  words”.  This  will  give  the  voice 
recognizer  a  greater  chance  of  matching  what  the  pilot 
says  with  the  proper  command  in  the  template.  Even 
with  just  limited  use  of  these  two  methods,  the  overall 
speech  recognition  error  rate  for  this  experiment  was 
held  to  around  1%! 

CONCLUSION 

In  this  experiment,  voice  control  was  found  to  be  a 
viable  control  method  for  single  seat  fighter  cockpits. 

On  the  plus  side,  for  most  tasks,  voice  provided  at  least 
equivalent  performance  to  the  manual  control  interface. 
However,  the  effectiveness  of  voice  will  also  depend 
upon  the  nature  of  the  tasks,  the  voice  implementation 
and  software  architecture,  and  the  context  in  which  it  is 
used.  \^ere  the  voice  implementation  takes  advantage 
of  well-learned  behaviors  (e.g.,  reading  strings  of 
alphanumerics),  or  can  reduce  the  number  of  command 
inputs  required  to  complete  a  task,  performance  is 
enhanced  and  workload  is  reduced.  Voice  also  has 
benefits  under  multitasking  conditions,  where  it  reduces 
resource  competition  by  offering  an  alternative  data 
entry/control  method  when  the  pilot’s  hands  and  eyes  are 
busy.  Voice  also  showed  significant  performance 
improvements  when  it  was  combined  with  automatic 
target  cuer  symbology  and  mechanized  to  enable  the 


pilot  to  directly  access  (target  cue  function)  and  select 
(target  designate  function)  critical  Information  (targets) 
on  a  display. 

On  the  minus  side,  a  voice  interface  has  the  potential  to 
degrade  performance  and  increase  workload  when  used 
to  replace  single  switch  hits  that  would  normally  be 
easily  accessible,  e.g.  via  HOTAS.  Also,  pilot 
comments  indicated  that  a  voice  interface  may  interfere 
with  radio  communications  monitoring. 

Even  with  the  potential  shortcomings  of  voice  control, 
pilots  unanimously  stated  that  they  would  want  speech 
recognition  capability  in  the  cockpit.  They  commented 
that  it  allowed  them  to  stay  head-up  and  “do  several 
things  at  once”  while  simultaneously  reducing 
workload — especially  during  keyboard  entry  tasks. 
Furthermore,  and  somewhat  unexpectedly,  pilot 
acceptance  of  voice  control  was  very  high  and  this 
acceptance  would  be  key  to  successfully  integrating  any 
new  technology  in  cockpits. 

As  with  any  conceptual  research,  the  results  are  highly 
dependent  upon  the  specific  evaluation  design  and 
technology  baseline,  and  the  conditions  under  which  the 
testing  occurred.  For  example,  voice  may  not  be 
advantageous  when  used  to  replace  discrete  switch  hits, 
but  may  be  highly  effective  when  combined  with  other 
advanced  technologies,  such  as  decision  aids  or 
automated  subsystems,  that  facilitate  the  use  of  voice 
“shortcuts”  (such  as  the  automatic  target  cuer  in  this 
experiment).  Similarly,  in  this  experiment,  voice  was 
used  in  an  artificial  environment  that  attempted  to 
induce  high  workload,  multitasking,  high  visual  and 
manual  task  loading  and/or  high  auditory  task  loading 
on  the  pilot.  The  results  may  have  been  significantly 
different  in  a  more  benign  task  environment  or  with 
different  workload  “inducers”.  These  factors  must  be 
taken  into  account  when  generalizing  the  results  of  this 
experiment  to  other  cockpit  applications. 
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ABSTRACT 

To  enable  aircraft  to  respond  to  time-critical  targets  during  a 
mission,  efforts  are  underway  to  develop  systems  that  provide 
pilots  access  to  imagery-based  information  from  offboard 
sources.  Previous  research  has  shown  that  annotated  imagery 
of  the  target  area  improves  target  recognition,  and  that  the 
usefulness  of  Imagery-Based  Target  Recognition  Aids 
(IBTRAs)  depends  on  factors  such  as  viewpoint,  image 
complexity,  and  coverage.  This  report  describes  a  study  that 
compared  the  utility  of  IBTRAs  generated  from  high- 
resolution  and  low-resolution  databases  and  identified  the 
impacts  of  each  on  operator  target  recognition  performance. 
Differences  in  performance  were  found  for  both  the  high-  and 
low-resolution  imagery  caused  by  variations  in  the  sensor 
distance  from  the  target  area  and  by  the  cross-range  extent  of 
the  image.  The  best  performance,  both  in  speed  and  accuracy, 
was  observed  when  the  cross-range  extent  of  the  IBTRA  was 
similar  to  that  of  the  sensor  image.  IBTRAs  developed  from 
high-resolution  data  provided  a  significant  increase  in  target 
recognition  accuracy  at  all  reference  image  cross-range 
extents  when  used  with  imagery  taken  close  to  the  target.  The 
operational  significance  of  these  findings  is  dependent  on  the 
scenario;  for  a  weapon  release  relatively  close  to  the  target, 
the  use  of  IBTRAs  generated  from  high-resolution  data  may 
be  of  greater  value  than  for  a  weapon  release  from  longer 
ranges.  A  model  is  presented  that  describes  target  recognition 
performance  as  a  function  of  the  cross-range  extent  of  the 
imagery  being  used. 


INTRODUCTION 

In  recent  years  efforts  have  begun  to  develop  various  systems 
which  would  provide  pilots  with  access  to  imagery-based 
information  from  either  ground  stations  or  other  airborne 
platforms.  As  currently  envisioned,  aircraft  would  be 
redirected  after  takeoff  in  response  to  newly  discovered  time- 
critical  threats.  The  planning  for  these  new  missions  would 
be  conducted  at  a  remote  ground  station  and,  upon 
completion  of  planning,  the  information  needed  for  a 
successful  attack  would  be  transmitted  to  the  pilot  in  the 
aircraft.  Upon  receiving  this  information,  the  pilot  would 
alter  course  and  attack  the  newly  assigned  target..  The  utility 
of  imagery  in  improving  target  recognition  was  verified  in  the 
research  described  in  reference  1.  Further  research 
(references  2,  3,  and  4)  however,  has  indicated  that  the 
usefulness  of  Imagery  Based  Target  Recognition  Aids 
(IBTRA’s)  depends  on  many  factors  including  viewpoint, 
image  complexity,  and  coverage. 

The  study  described  in  reference  4  demonstrated  that  one  of 
the  critical  parameters  affecting  the  utility  of  IBTRA’s  was 
the  area  of  coverage  provided  in  the  image.  The  IBTRA’s 
used  in  reference  4  were  generated  from  an  image  database 
with  a  resolution  of  30  feet  per  datum.  However, 
examination  of  the  IBTRA’s  used  in  that  study  revealed 
significant  degradation  in  image  quality  when  producing 
IBTRA’s  of  limited  cross-range  extent.  While  unavoidable, 
this  raised  the  question  of  whether  generating  the  IBTRA’s 
from  a  high  resolution  database  would  significantly  improve 
performance.  The  purpose  of  the  study  described  in  this 
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report  was  to  compare  the  utility  of  IBTRA’s  generated  from 
high  resolution  sources  with  those  generated  from  low 
resolution  sources  and  identify  the  impacts  which  these 
sources  would  have  on  operator  target  recognition 
performance.  There  are  currently  two  classes  of  “high 
resolution”  databases  available.  The  first  includes  databases 
containing  imagery  which  is  created  from  high  resolution 
sources.  The  second  type  are  collected  from  sources  with 
lower  resolutions  and  then  post  processed  to  produce  a 
database  which  contains  interpolated  values  for  points  not 
contained  in  the  original  imagery.  The  imagery  contained  in 
these  “interpolated  databases”  are  of  inherently  lower  and 
users  of  digital  data  must  take  care  to  ensure  that  the  type  of 
database  selected  will  meet  the  needs  of  the  intended 
application. 

METHOD 

Participants 

The  participants  for  this  experiment  consisted  of  thirty  male 
and  female  employees  of  the  NAWCWPNS. 

Equipment 

The  experiment  console  for  this  study  was  implemented  on  a 
166  MHz  Pentium  computer.  The  imagery  was  stored  in 
digital  format  on  the  local  hard  disk  and  was  displayed  on  a 
Mitac  17-inch  monitor.  The  monitor  resolution  used  for  the 
experiment  console  was  1280  pixels  wide  by  1024  pixels 
high.  This  resolution  was  chosen  because  it  provided  a  pixel 
density  (number  of  pixels  per  inch)  close  to  that  of  a  tactical 
aircraft  display. 

Stimuli 

FUR  Imagery 

The  sensor  imagery  used  in  this  study  was  culled  from 
videotapes  recorded  in  the  local  area  during  exercise  and 
training  flights  using  a  Forward  Looking  Infrared  (FLIR) 
system.  The  FLIR  imagery  was  categorized  as  either  “close”, 
“medium”,  or  “far”  based  on  the  range  from  the  FLIR  sensor 
to  the  target.  Since  the  actual  ranges  for  the  FLIR  imagery 
used  in  this  study  were  unavailable,  the  cross-range  extent  of 
each  image  was  measured  directly  from  the  image.  The  term 
“cross-range  extent”  as  used  in  this  report  refers  to  the 
distance  between  the  two  points  on  the  ground  displayed  at 
the  opposite  center  points  of  the  vertical  sides  of  the  image. 
The  “close”  FLIR  imagery  had  cross-range  extents  varying 
from  0.5  miles  to  0.75  miles.  The  “medium”  images  included 
cross-range  extents  from  1  miles  to  2  miles.  The  cross-range 
extents  of  the  “far”  images  varied  from  3  miles  to  4  miles. 
FLIR  images  with  cross-range  extents  which  did  not  fit  one  of 
these  categorizations  were  not  used. 


Reference  Imagery 

The  low  resolution  reference  scenes  were  extracted  from 
commercially  available  SPOT  imagery.  This  product  has  a 
resolution  of  10  meters  per  datum.  For  each  target  used  in  the 
experiment,  a  plan  view  image  was  generated  for  each  of  five 
cross-range  extents;  .25  miles,  .5  miles,  1.5  miles,  3.5  miles, 
and  5  miles.  The  high  resolution  reference  scenes  were 
extracted  from  United  States  Geologic  Survey  (USGS)  aerial 
reconnaissance  imagery.  This  is  a  film  product  and  was 
digitized  by  use  of  a  high  resolution  drum  scanner.  The 
resulting  digitized  imagery  has  a  resolution  of  3  feet  per 
datum.  For  each  target  used  in  the  experiment,  a  plan  view 
image  was  generated  for  each  of  four  cross-range  extents 
were:  0.125  miles,  .25  miles,  .5  miles,  and  1.5  miles.  The 
resolution  of  the  reference  images  was  500  pixels  vertically 
and  horizontally  and  the  images  were  displayed  at  a 
resolution  of  approximately  100  pixels  per  display  inch. 

Procedure 

Description  of  the  Experimental  Task 

The  task  during  the  experiment  trials  consisted  of  a  target 
recognition  task.  In  each  trial  for  this  task,  the  subjects  were 
presented  with  two  images;  a  FLIR  image  (either  close, 
medium,  or  far)  and  a  reference  image.  In  each  reference 
image  a  target  was  indicated  by  a  box  circumscribing  the 
object,  which  the  subjects  were  required  to  locate  in  the  FLIR 
image.  The  subjects  indicated  their  response  in  this  task  by 
placing  the  mouse  cursor  over  the  location  of  their  choosing 
in  the  FLIR  image  and  pressing  the  left  mouse  button.  The 
subjects  were  allowed  to  take  as  much  time  as  they  needed  to 
designate  the  target.  The  data  collected  for  this  task  consisted 
of  the  location  of  the  mouse  when  the  subjects  indicated  their 
choice  and  the  time  which  had  elapsed  since  the  onset  of  the 
images. 

Design 

This  study  was  structured  as  three  separate  sub-experiments. 
The  first  consisted  of  two  fully  crossed  within  factors;  FLIR 
sensor  distance  with  3  levels  (close,  medium,  and  far)  and 
low  resolution  reference  image  extent  with  5  levels  (.25,  .5, 
1.5,  3.5,  and  5  miles).  The  second  consisted  of  two  within 
fully  crossed  factors;  FLIR  sensor  distance  with  the  same  3 
levels  and  high  resolution  reference  image  cross-range  extent 
with  four  levels  (.0125,  .25,  .5,  and  1.5  miles).  The  third 
consisted  of  three  fully  crossed  factors;  one  between  factor, 
database  resolution,  with  2  levels  (high  and  low),  and  two 
within  factors;  FLIR  sensor  distance,  with  3  levels  (close, 
medium,  and  far)  and  reference  image  cross-range  extent, 
with  three  levels  (.25,  .5,  and  1.5  miles).  The  additional  FLIR 
sensor  distance  for  the  high  resolution  cross-range  extents 
(0.125  miles)  was  included  to  permit  examination  of  any 
shifts  in  performance  which  might  be  observed  in  relation  to 
the  use  of  the  high  resolution  images. 
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RESULTS 

Analysis  of  the  Low  Resolution  Data 

Speed 

The  response  times  for  the  low  resolution  cases  were 
tabulated  and  subjected  to  an  Analysis  of  Variance 
(ANOVA).  The  results  of  the  analysis  indicated  that  there 
were  significant  main  effects  for  both  FLIR  sensor  distance 
(FSR)  (F2,28  =  6.65,  p  <  0.0043)  and  reference  image  cross¬ 
range  extent  (CRE)  (F4,56  =  3.25,  p  <  0.0181).  There  was 
also  a  significant  interaction  between  FSR  and  CRE  (F8,l  12 
=  3.46,  p  <  0.0013).  Further  analysis  indicated  that  the  Far 
FSR  condition  was  significantly  different  from  the  Close  and 
Medium  FSR  conditions  (Fl,14  =  79.25,  p  <  0.0009)  but  that 
the  Close  and  Medium  conditions  were  not  significantly 
different  from  each  other.  Analysis  also  indicated  that 
response  times  for  the  0.5  mile  and  5.0  mile  CRE  conditions 
were  significantly  different  from  the  response  times  for  the 
0.25  mile,  1.5  mile,  and  3.5  mile  CRE  conditions  (FI, 14  = 
8.04,  p  <  0.0132).  The  results  of  this  analysis  are  presented  in 
figure  1. 
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Figure  1 .  Response  Time  as  a  function  of  Reference 
Image  Cross-Range  Extent  and  FLIR  Sensor  Distance 
for  the  Low  Resolution  Data 

Accuracy 

The  percentages  of  eorrect  responses  for  the  low  resolution 
cases  were  tabulated  and  subjected  to  an  ANOVA.  The 
results  indicated  that  there  were  significant  main  effects  for 
FSR  (F2,28  =  15.98,  p  >  0.00001),  CRE  (F4,56  =  6.24,  p  < 
0.0003),  and  a  significant  interaction  between  the  two 
(F8,112  =  4.14,  p  <  0.0002).  Contrasts  indicated  that  the 
accuracies  for  the  Close  and  Medium  FSR  conditions  were 
not  significantly  different  but  that  the  percentages  of  correct 
responses  for  these  two  condition  were  significantly  different 
than  those  for  the  Far  FSR  condition  (FI,  14  =  34.16,  p  < 


0.00001).  Further  analysis  indicated  that  the  observed 
accuracies  divided  into  three  groups  with  respect  to  CRE. 
The  first  group  included  the  0.25  mile  and  3.5  mile  CRE 
conditions  which  differed  significantly  from  the  5.0  mile 
CRE  condition  (FI, 14  =  4.78,  p  <  0.0462)  and  from  the  group 
containing  the  0.5  mile  and  1.5  mile  CRE  conditions  (FI, 14  = 
10.42,  p  <  0.0061).  The  results  of  this  analysis  are  presented 
in  figure  2. 
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Figure  2.  Percentage  Correct  Responses  as  a  function 
of  Reference  Image  Cross-Range  Extent  and  FLIR 
Sensor  Distance  for  the  Low  Resolution  Data 

Analysis  of  the  High  Resolution  Data 

Speed 

The  response  times  for  the  high  resolution  cases  were 
tabulated  and  subjected  to  an  ANOVA.  The  results  of  this 
analysis  indicated  that  there  were  main  effects  for  FSR  (F2,13 
=  12.61,  p  <  0.0001)  and  CRE  (F3,42  =  4.28,  p  <  0.0101)  and 
a  significant  interaction  between  FSR  and  CRE  (F6,84  = 
2.32,  p  <  0.0404).  Contrasts  indicated  that  there  was  a 
significant  difference  between  the  response  times  for  the 
Close  and  Medium  FSR  conditions  (FI,  14  =  20.20,  p  < 
0.0005)  but  only  a  marginal  difference  between  the  response 
times  for  the  Medium  and  Far  conditions  (Fl,14  =  4.42,  p  < 
0.0541).  Further  analysis  indicated  that  the  response  times 
divided  into  two  groups  with  respect  to  the  CRE  conditions. 
Significant  differences  were  found  between  the  response 
times  for  the  first  group  (which  consisted  of  the  0.125  mile 
and  0.25  miles  CRE  conditions)  and  the  second  (which 
consisted  of  the  0.5  mile  and  1.5  miles  CRE  conditions) 
(FI, 14  =  7.15,  p  <  0.0182).  The  results  of  this  analysis  are 
presented  in  figure  3. 
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Figure  3.  Response  Time  as  a  function  of  Reference 
Image  Cross-Range  Extent  and  FLIR  Sensor  Distance 
for  the  High  Resolution  Data 


Accuracy 

The  accuracies  for  the  high  resolution  cases  were  tabulated 
and  subjected  to  an  ANOVA.  The  results  indicated  that  there 
were  significant  main  effects  for  FSR  (F2,28  -  145,  p  < 
0.00001)  and  CRE  (F3,42  =  3.07,  p  <  0.0382)  and  a 
significant  interaction  between  the  two  (F6,84  =  5.54,  p  < 
0.0001).  Further  analysis  indicated  that  there  were  significant 
differences  between  the  accuracies  for  all  three  FSR 
conditions  (Close  v.  Medium:  FI, 14  =  40.19,  p  <  0.00001 
Medium  v.  Far:  FI, 14  =  127.88,  p  <  0.00001),  Contrasts  also 
indicated  that  the  accuracies  for  the  0.125  mile  CRE 
condition  were  significantly  different  from  the  rest  (FI, 14  = 
4.78,  p  <  0.0462),  but  that  the  accuracies  for  the  0,25  mile, 
0.5  mile,  and  1.5  mile  conditions  did  not  significantly  differ 
from  each  other.  The  results  of  this  analysis  are  presented  in 
figure  4. 


Figure  4.  Percentage  of  Correct  Responses  as  a 
Function  of  Reference  Image  Cross-Range  Extent  and 
FLIR  Sensor  Distance  for  the  High  Resolution  Data 


Comparison  of  the  High  and  Low  Resolution  Data 

The  following  analyses  were  conducted  on  those  FSR  and 
CRE  conditions  which  were  observed  for  both  the  high  and 
low  resolution  conditions.  The  common  FSR  conditions 
included  Close,  Medium,  and  Far.  The  common  CRE 
conditions  included  0.25  miles,  0.5  miles,  and  1.5  miles. 

Speed 

The  response  times  for  the  common  conditions  for  the  high 
and  low  resolution  cases  were  tabulated  and  subjected  to  an 
ANOVA.  The  results  of  this  analysis  indicated  that  there 
were  main  effects  for  FSR  (F2,56  =27.69,  p  <  0.00001)  and 
CRE  (F2,56  =  4.34,  p  <  0.0176).  There  were  also  two 
significant  two-way  interactions;  Resolution  (RES)  x  CRE 
(F2,56  =  5.59,  p  <  0.0061)  and  FSR  x  CRE  (F4,l  12  =  5.20,  p 

<  0.0007),  and  a  marginally  significant  three-way  interaction: 
RES  X  FSR  X  CRE  (F4,112  =  2.42,  p  <  0.0522).  Contrasts 
indicated  that  there  was  a  significant  effect  for  CRE  at  the 
Medium  and  Far  levels  of  FSR  (Medium:  F2,56  =  6,15,  p  < 
0.0039,  Far:  F2,56  =  5.85,  p  <  0.0049)  and  a  significant 
interaction  between  RES  and  CRE  at  the  Far  FSR  level 
(F2,56  =  7.83,  p  <  0.0010).  Further  analysis  indicated  that 
there  were  significant  effects  for  FSR  at  each  level  of  CRE 
(0.25  mile:  F2,56  =  7.96,  p  <  0.0009,  0.5  mile:  F2,56  =  16.13, 
p  <  0.00001,  1.5  mile:  F2,56  =  7.14,  p  <  0.0017)  and  a 
significant  interaction  between  FSR  and  RES  at  the  0.25  mile 
level  for  CRE  (F2,56  =  3.21,  p  <  0,0481).  Simple  effects 
comparisons  indicated  that  there  was  a  significant  difference 
between  the  accuracies  observed  for  the  High  and  Low  RES 
conditions  at  the  0.25  mile  level  of  CRE  for  the  Far  FSR 
condition  (FI, 28  =  8.03,  p  <  0,0084).  Figures  5  through  7 
present  the  comparisons  of  the  high  and  low  resolution  data 
for  each  of  levels  of  FSR.  The  mean  values  of  all  of  the 
levels  of  CRE  for  both  the  high  and  low  resolution  cases  have 
been  presented  in  these  figures  to  provide  the  reader  with  a 
better  understanding  of  the  data.  The  reader  should  keep  in 
mind,  however,  that  the  analysis  comparing  the  high  and  low 
resolution  cases  involved  only  those  levels  which  the  two  had 
in  common  (0.25  mile,  0.5  mile,  and  1.5  mile). 

Accuracy 

The  percentages  of  correct  responses  for  the  common 
conditions  for  the  high  and  low  resolution  cases  were 
tabulated  and  subjected  to  an  ANOVA.  The  results  of  this 
analysis  indicated  that  there  were  main  effects  for  RES  (Fl,28 
=  9.90,  p  <  0.0038),  FSR  (F2,56  =  73.51,  p  <  0.00001),  and 
CRE  (F2,56  =  4.47,  p  <  -.0158).  There  were  also  two 
significant  two-way  interactions;  RES  x  FSR  (F2,56  =  5.31,  p 

<  0.0077)  and  FSR  x  CRE  (F4,112  =  7.52,  p  <  0.00001),  and 
a  significant  three-way  interaction:  RES  x  FSR  x  CRE 
(F4,112  =  2.79,  p  <  0.0298).  Contrasts  indicated  that  there 
was  a  significant  effect  for  CRE  at  each  level  of  FSR  (Close: 
F2,56  =  7.48,  p  <0.0013,  Medium:  F2,56  =  4.85, 
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Figure  5.  Response  Time  as  a  function  of  Reference 
Image  Cross-Range  Extent  and  Source  Database 
Resolution  for  the  Close  FLIR  Sensor  Distance. 
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Figure  6.  Response  Time  as  a  function  of  Reference 
Image  Cross-Range  Extent  and  Source  Database 
Resolution  for  the  Medium  FLIR  Sensor  Distance. 

p  <  0.0114,  Far:  F2,56  =  7.99,  p  <  0.0009)  and  a  significant 
interaction  between  RES  and  CRE  at  the  Far  FSR  level 
(F2,56  =  3.50,  p  <  0.0371).  Further  analysis  indicated  that 
there  were  significant  effects  for  FSR  at  each  level  of  CRE 
(0.25  mile:  F2,56  =  52.13,  p  <  0.00001,  0.5  mile:  F2,56  = 
56.09,  p  <  0.00001,  1.5  mile:  F2,56  =  9.07,  p  <  0.0004)  and  a 
significant  interaction  between  FSR  and  RES  at  the  0.25  mile 
level  for  CRE  (F2,56  =  10.83,  p  <  0.0001).  Simple  effects 
comparisons  indicated  that  there  was  asignificant  difference 
between  the  accuracies  observed  for  the  High  and  Low  RES 
conditions  at  all  levels  of  CRE  for  the  Close  FSR  condition 
(0.25  mile:  FI, 28  =  23.04,  p  <  0.00001,  0.5  mile:  Fl,28  = 
7.78,  p  <  0.0094,  1.5  mile:  FI, 28  =  5.65,  p  <  0.0245),  at  the 
0.25  mile  level  of  CRE  for  the  medium  FSR  condition  (FI, 28 
=  5.50,  p  <  0.0264),  and  at  the  1.5  mile  level  of  CRE  for  the 


Far  FSR  condition  (Fl,28  =  7.29,  p  <  0.0116).  Figures  8 
through  10  present  the  comparisons  of  the  high  and  low 
resolution  data  for  each  of  levels  of  FSR.  The  mean  values 
of  all  of  the  levels  of  CRE  for  both  the  high  and  low 
resolution  cases  have  been  presented  in  these  figures  to 
provide  the  reader  with  a  better  understanding  of  the  data. 
The  reader  should  keep  in  mind,  however,  that  the  analysis 
comparing  the  high  and  low  resolution  cases  involved  only 
those  levels  which  the  two  had  in  common  (0.25  mile,  0.5 
mile,  and  1.5  mile). 
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Figure  7.  Response  Time  as  a  function  of  Reference 
Image  Cross-Range  Extent  and  Source  Database 
Resolution  for  the  Far  FLIR  Sensor  Distance. 
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Figure  8.  Percentage  of  Correct  Responses  as  a 
function  of  Reference  Image  Cross-Range  Extent  and 
Source  Database  Resolution  for  the  Close  FLIR  Sensor 
Distance. 

SUMMARY 

The  findings  from  this  study  indicate  that  operator  target 
recognition  performance  using  IBTRA’s  developed  from  both 
high  and  low  resolution  sources  corresponds  fairly  well  to  the 
model  described  in  the  early  sections  of  this  report.  The  best 
performance,  both  in  speed  and  accuracy,  was  observed  when 
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Figure  9.  Percentage  of  Correct  Responses  as  a 
function  of  Reference  Image  Cross-Range  Extent  and 
Source  Database  Resolution  for  the  Medium  FLIR 
Sensor  Distance. 
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Figure  10.  Percentage  of  Correct  Responses  as  a 
function  of  Reference  Image  Cross-Range  Extent  and 
Source  Database  Resolution  for  the  Far  FLIR  Sensor 
Distance. 

the  cross-range  extent  of  the  auxiliary  image  (the  IBTRA) 
was  similar  to  that  of  the  FLIR  sensor  image.  As  expected, 
the  peak  performance  for  the  low  resolution  IBTRA’s  was 
better  for  conditions  where  the  cross-range  extent  of  the 
images  was  smaller  and  degraded  gradually  as  the  cross-range 
extents  increased.  The  shape  of  the  curves  associated  with  the 
high  resolution  IBTRA’s  would  seem  to  indicate  that,  had 
higher  cross-range  extents  been  examined  in  this  study,  a 
similar  pattern  might  have  been  observed.  However,  further 
research  would  be  required  to  substantiate  this.  Overall,  the 
correspondence  between  the  model  and  observed 
performance  appears  to  be  closer  for  the  observed  accuracies 
than  for  the  observed  response  times.  While  the  model 
predicts  a  relatively  flat,  wide  central  region  where  operator 
performance  is  relatively  unaffected  by  either  lack  of  context 
or  resolution,  the  actual  regions  observed  in  the  data  appeared 
to  be  quite  narrow. 


IBTRA’s  developed  from  high  resolution  data  provided  a 
significant  increase  in  target  recognition  accuracy  at  all 
reference  image  cross-range  extents  ranges  when  used  with 
FLIR  imagery  taken  close  to  the  target.  However,  this  benefit 
only  appeared  at  the  0.25  mile  IBTRA  cross-range  extent  for 
the  medium  sensor  distance,  and  at  the  1.5  miles  IBTRA 
cross-range  extent  for  the  far  sensor  range.  There  appeared  to 
be  no  affect  on  response  time  due  to  the  use  of  high 
resolution  data  except  at  0.25  mile  IBTRA  cross-range 
extents  with  far  sensor  ranges  where  operators  took 
significantly  longer  to  use  IBTRA’s  derived  from  high 
resolution  data  than  to  use  IBTRA’s  derived  from  low 
resolution  data.  This  may  have  been  due  to  a  tendency  on  the 
operators  to  spend  time  examining  the  detail  in  the  high 
resolution  aids  which  was  unavailable  for  examination  in  the 
low  resolution  aids.  Another  explanation  may  be  that  the 
subjects  may  have  more  readily  “given  up”  when  presented 
with  a  severely  degraded  low  resolution  image  as  a  reference. 
The  extra  time  spent  on  using  this  imagery,  though,  did  not 
yield  a  significant  increase  in  accuracy. 

The  operational  significance  of  these  findings  is  dependent  on 
the  scenario  under  discussion.  If  the  mission  in  question 
involves  a  weapon  release  relatively  close  to  the  target,  then 
the  use  of  IBTRA’s  constructed  from  a  high  resolution  source 
may  be  of  value.  If,  however,  the  weapons  involved  will  be 
released  from  longer  ranges,  high  resolution  aid  may  not  be 
as  useful.  What  ranges  constitute  “relatively  close”  or 
“longer”  would  depend  on  the  field  of  view  of  the  sensor  in 
use.  For  example,  if  the  sensor  has  a  5  degree  field  of  view, 
then  IBTRA’s  constructed  from  higher  resolutions  might  be 
useful  at  ranges  within  about  15-20  miles  from  the  target. 
However,  if  the  field  of  view  of  the  sensor  was  30  degrees, 
these  IBTRA’s  might  not  be  useful  until  the  pilot  is  within  1- 
2  miles  of  the  target.  An  interesting  point  which  is  evident 
from  examination  of  figures  30  -  35  is  that,  when  operator 
accuracy  was  better  using  a  high  resolution  aid,  the  region  of 
best  performance  did  not  change.  This  would  lead  to  the 
conclusion  that,  if  designers  were  restricted  to  the  choice  of  a 
single  cross-range  extent  for  generating  both  high  and  low 
resolution  aids,  then  a  cross-range  extent  close  to  that  of  the 
sensor  would  produce  the  best  performance.  Also,  if 
designers  were  limited  to  a  single  cross-range  extent  for  all 
IBTRA’s  regardless  of  the  range  to  target,  then  IBTRA’s  with 
a  cross-range  extent  of  between  1.0  to  3.0  miles  would 
probably  be  best. 
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